Obviously there will be some sort of hit, but if things are indexed properly, the hit will be small.
If you add a child record, SQL server has to make sure the parent exists... as this is a lookup on the primary key of the parent, it will already have an index to support this lookup.
If you try to delete a parent record, SQL Server has to make sure there are no child records referencing it. If the foreign key column(s) in the child table aren't indexed, SQL Server will have to scan the whole table to check for the existence of a child record... then you will get a performance hit.