How to conditionally copy a substring into a new column of a pandas dataframe?

You can use pd.Series.str.extract:

In [737]: df
Out[737]: 
         A                                   B
0    VALID       asdfafX'XextractthisY'Yeaaadf
1  INVALID         secondrowX'XsubtextY'Yelakj
2    VALID  secondrowX'XextractthistooY'Yelakj

In [745]: df['C'] = df[df.A == 'VALID'].B.str.extract("(?<=X'X)(.*?)(?=Y'Y)", expand=False)

In [746]: df
Out[746]: 
         A                                   B               C
0    VALID       asdfafX'XextractthisY'Yeaaadf     extractthis
1  INVALID         secondrowX'XsubtextY'Yelakj             NaN
2    VALID  secondrowX'XextractthistooY'Yelakj  extractthistoo

The regex pattern is:

(?<=X'X)(.*?)(?=Y'Y)

• (?<=X'X) is a lookbehind for X'X

• (.*?) matches everything between the lookbehind and lookahead

• (?=Y'Y) is a lookahead for Y'Y