The castling issue is a good example of the different design philosophies that are possible, so let me respond to that first. If an engine supports Capablanca Chess, it is unlikely to support flexible castling. The only reason why the author would care to support such castling is that he is aware of Scholbook Chess, and wants the engine to be able to play that too. But if he wants that, why would he want the user to alter the start position (through setboard) and the castling rules (through an option) separately? It would be much simpler, for him as well as the user, to just add Schoolbook as an additional variant the engine can play. If there are other 10x8 variants that use flexible castling, but have a different start position from Schoolbook, these could then be played as sub-variants of Schoolbook (through setboard). Variants without castling are sub-variants anyway, as you can simply specify a start position without castling rights.

Having a castling option is a step towards defining the variant for the engine through option settings. If you have an option for castling, then why not one for how the super-piece moves (Queen / Chancellor / Archbishop)? Or options for how any piece moves? Or how large the promotion zone is? Or what you can promote to? (To play Janus as sub-variant of Capablanca.) This would all be possible, of course; it would converge on a design where the GUI manages all game descriptions (be it hard-coded or through a configuration file that the user can edit), and then instructs an engine how to play it.

This is not how existing multi-variant engines such as Fairy-Max, Sjaak and Nebiyu work, however. These manage the file with game definitions themselves. Which seems the logical way to do it, as then the file can also contain engine-specific info. (Like which pieces have mating potential, value of pair bonuses etc.)

Because of the way XBoard started (as merely a graphical front-end for an already complete ASCII-based user interface) things tend to be 'engine-centered' in WB protocol. From the early days on XBoard could already be used to play variants (as long as they were 8x8 with no more than 6 piece types, such as Shatranj) by switching legality testing off. (In fact in the earliest version it could not even be switched on, as legality testing was not implemented at all. Even when I first started working on WinBoard myself in 2008 it did not know about castling rights, and would always accept any castling.) Historically it was up to the engine to refuse moves it considered illegal, on which the GUI would retract the move and relay the info to the user. Downsite of this is that the GUI cannot indicate target squares of a move when pieces might move different from normal, and that it will generate crappy SAN (with missing or redundant disambiguation, but which it would still understand itself on reading it back).

I generalized the 'legality off' capabilities a bit by attaching a meaning to some otherwise unused combinations of mouse clicks; e.g. any sideway move of a King by more than 2 spaces is now interpreted as an attempt to castle with the corner piece in that direction (whatever it is), while dragging the King on top of the corner piece would be castling with it where the King steps one. (In all cases the corner piece ends up on the other side of the King.) Unless Fischer castling is specified by a GUI option (which by default is set for FRC and CRC), then capturing an own backrank piece with the King indicates castling with that. This covered all cases I was aware of, so the GUI doesn't really have to know the game rules to interpret the user move; assuming that what the user tries is legal, it will interpret it correctly. The engine watches over the latter, by forcing the GUI to take back whatever illegal thing the user might have done.

For better GUI performance w.r.t. SAN and target-square highlighting, however, it is essential that the GUI is also aware of the rules. Logically there are 3 ways to accomplish this:

1. Both the GUI and the engine are programmed / configured for the same rules independently.
2. The Engine transfers the rules to the GUI
3. The GUI transfers the rules to the engine

Your castling option is an example of the latter; through engine-defined options any info can in principle be transferred in this direction, but some standardization would have to be developed to make it possible for the GUI to do this automatically (rather than having the user decide on a per-engine basis what is necessary). For the start position the protocol has the setboard (or edit) command. For case (2) I extended the protocol with a 'setup' command as a reverse setboard, (plus a list of participating pieces) and 'piece' commands for defining piece moves.

Fairy-Max and Sjaak || use method (2); their game-definition file specifies how castling is to be performed, and (if this is non-standard) they inform the GUI of it as part of a King-move definition, e.g. "piece K KisO4", where the isO4 is the XBetza notation of a castling where the King moves 4 squares. Through this info XBoard itself can then already refuse any attempt of the user to move the King other than 1 step or slide it 4 (if still virgin etc.) without first consulting the engine. (And does this by not even highlighting other King destinations, and in general refusing any move to a non-highlighted square.)

I guess we agree that method (1) is undesirable; this was basically the situation that existed before introduction of the setup and piece commands, where the user had to provide a FEN (and even in an indirect way, as a .fen file with a GUI option referring to it) and a -pieceToCharTable option to specify the FEN's meaning, and switching off legality testing when standard glyphs were used to represent exotic pieces not in the standard set. For the case of castling it seems to me that method (2) is more direct than (3). In all cases the engine would have to announce something to the GUI at startup, either that it has a castling option that features 'flexible' as a choice, or just that it supports Schoolbook. In the case of the option the GUI will have to recognize it, look up in a list of known variants which variants would need any of the available settings for that option, and enable selection of those variants by the user (if their other requirements are also satisfied). In the other case it can just enable selection of 'Schoolbook' without having to worry about anything, or even know what 'Schoolbook' means. When the user decides to select Schoolbook, in the case of the option the GUI would now have to set the castling option in the engine to the value it knew to be required for Schoolbook, as well as enable that type of castling in its own move generator used for legality testing / square highlighting / SAN parsing and generation. In method (2) it would just configure its move generator likewise because the engine tells it so through a piece command. Again, it did not need to have any pre-programmed knowledge for that. The 'piece' command can be considered in this case just as a (somewhat cryptic) command to set a GUI option for the type of castling. This is much less of a hassle for the GUI.

I am perfectly happy with XBoard being a 'dumb' GUI, which leaves the engine in charge. But if you want ChessV to be a GUI that knows about many variants, and makes conclusions about which variants not explicitly supported by the engine can still be played as sub-variant of some other, supported variant, method (2) would still allow you to do it. In all cases the GUI would look which variants the engine supports, and what aspects of variants the engine allows to be changed in general (startposition by setboard, possibly other options), and then can run through its list of known variants to see which can be run as sub-variant. It doesn't really matter if a 10x8 variant has to be run as Capablanca + setboard + castling=flexible or through Schoolbook + setboard. If you want the GUI to have intrinsic knowledge of what Schoolbook is, it can simply ignore setup or piece commands the engine sends to specify start position and castling type.