It's a backreference to the first capture group, which is denoted by the first set of parenthesis, e.x.

    (.*)+\1

would match "ABAB", or "ABCABC", etc.

"\2" would match the second capture group, e.x.

    (.*)(.*)\2\1

would match "ABBA", or "AABBBBAA", etc.

so in your first example, "AB" was in the capture group and the \1 repeats it?

Correct. The following

    (.*)\1+

Would also match "ABABAB", for example.

Speaking of that puzzle how come [COBRA]+ and (.)+\1 give back result 'O'? Why not C or B or A?

(.)+\1 means the last letter has to be equal to the first letter.

(AB|O|OR)+ dictates that the last letter of the first row has to be A or O

[^ABRC]+ makes it so that the lower right square can't be B therefore the upper right square can't be A

Look at the Help button (top right); that explains all their syntax.

The help button doesn't really explain that point ( I don't think) at least not in a very clear way for someone inexperienced with regex