In Classic Mode Authentication, IIS relies on Active Directory to perform authentication.
In Claims Based Authentication, the Security Token Service (STS) relies on a trusted authority to perform authentication. The authority can be Active Directory provider or other providers.
If the Claim Based Authentication uses the Windows authentication provider, the Security Token Service (STS) performs essentially the same function as IIS does in Classic Authentiction.
Form-Based Authentication with Windows Authentication Provider

Create a TestOU in Active Directory and move users and groups under it.

SharePoint 2010 is set as a farm, using SQL Server 2008 R2 at King member computer.
Create a Web Application--Claims Based Authentication--Enable Forms Based Authentication (FBA)

Open "Internet Information Service (IIS) Manager"
Right Click "SharePoint Central Administration"--Select "Explore"
Open Web.config file in notepad
Find <system.web>
Paste the follow code directly below it:
<membership defaultProvider="AspNetSqlMembershipProvider">
<providers>
<add name="LdapMember"
type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
server="k1.carteasy.com"
port="389"
useSSL="false"
userDNAttribute="distinguishedName"
userNameAttribute="sAMAccountName"
userContainer="OU=testou,DC=carteasy,DC=com"
userObjectClass="person"
userFilter="(ObjectClass=person)"
scope="Subtree"
otherRequiredUserAttributes="sn,givenname,cn" />
</providers>
</membership>
<roleManager enabled="true" defaultProvider="AspNetWindowsTokenRoleProvider" >
<providers>
<add name="LdapRole"
type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
server="K1.carteasy.com"
port="389"
useSSL="false"
groupContainer="OU=TestOU,DC=carteasy,DC=com"
groupNameAttribute="cn"
groupNameAlternateSearchAttribute="samAccountName"
groupMemberAttribute="member"
userNameAttribute="sAMAccountName"
dnAttribute="distinguishedName"
groupFilter="((ObjectClass=group)"
userFilter="((ObjectClass=person)"
scope="Subtree" />
</providers>
</roleManager>

Remove other <membership> and <rolemanager> entries.
Add two entries to <PeoplePickerWildcards> block
<add key="LdapMember" value="*" />
<add key="LdapRole" value="*" />

Open "Internet Information Service (IIS) Manager"
Expand "SharePoint Web Services"
Right click "SecurityTokenServiceApplication" and Select "Explorer"
Open "Web.config" in notepad
Find </system.net>
Add the following code directly below it
<system.web>
<membership>
<providers>
<add name="LdapMember"
type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
server="K1.carteasy.com"
port="389"
useSSL="false"
userDNAttribute="distinguishedName"
userNameAttribute="sAMAccountName"
userContainer="OU=testOU,DC=carteasy,DC=com"
userObjectClass="person"
userFilter="(&(ObjectClass=person))"
scope="Subtree"
otherRequiredUserAttributes="sn,givenname,cn" />
</providers>
</membership>
<roleManager enabled="true" >
<providers>
<add name="LdapRole"
type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
server="K1.carteasy.com"
port="389"
useSSL="false"
groupContainer="OU=testOU,DC=carteasy,DC=com"
groupNameAttribute="cn"
groupNameAlternateSearchAttribute="samAccountName"
groupMemberAttribute="member"
userNameAttribute="sAMAccountName"
dnAttribute="distinguishedName"
groupFilter="(&(ObjectClass=group))"
userFilter="(&(ObjectClass=person))"
scope="Subtree" />
</providers>
</roleManager>
</system.web>

Open "Internet Information Service (IIS) Manager"

Open "Web.config" in notepad
paste the following code
<add name="LdapMember" type="Microsoft.Office.Server.Security.LdapMembershipProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" server="k1.carteasy.com" port="389" useSSL="false" userDNAttribute="distinguishedName" userNameAttribute="sAMAccountName" userContainer="OU=testOU,DC=carteasy,DC=com" userObjectClass="person" userFilter="(&(ObjectClass=person))" scope="Subtree" otherRequiredUserAttributes="sn,givenname,cn" />
<add name="LdapRole" type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" server="k1.carteasy.com" port="389" useSSL="false" groupContainer="OU=testOU,DC=carteasy,DC=com" groupNameAttribute="cn" groupNameAlternateSearchAttribute="samAccountName" groupMemberAttribute="member" userNameAttribute="sAMAccountName" dnAttribute="distinguishedName" groupFilter="(&(ObjectClass=group))" userFilter="(&(ObjectClass=person))" scope="Subtree" />

Add a user policy to the Web Application http://k2:2198
Central Administration/Application Management/Manage Web Applications
Select http://k2:2198



http://k2:2198/

Site Permissions
Add Nicole and Sam to Members group
Please select the user from Forms Auth box

Then, Both Nicole and Sam can sign in to the web site by Forms Authentication.
In the three web.config, K1.carteasy.com is used because my domain has only one domain controller.
It is better to use server=carteasy.com if there are many domain controllers in your domain. For example,
<add name="LdapRole" type="Microsoft.Office.Server.Security.LdapRoleProvider, Microsoft.Office.Server, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" server="carteasy.com" port="389" useSSL="false" groupContainer="OU=testOU,DC=carteasy,DC=com" groupNameAttribute="cn" groupNameAlternateSearchAttribute="samAccountName" groupMemberAttribute="member" userNameAttribute="sAMAccountName" dnAttribute="distinguishedName" groupFilter="(&(ObjectClass=group))" userFilter="(&(ObjectClass=person))" scope="Subtree" />