xdsclient: stop batching writes on the ADS stream (#8627)
Fixes https://github.com/grpc/grpc-go/issues/8125
#### The original race in the xDS client:
- Resource watch is cancelled by the user of the xdsClient (e.g.
xdsResolver)
- xdsClient removes the resource from its cache and queues an
unsubscribe request to the ADS stream.
- A watch for the same resource is registered immediately, and the
xdsClient instructs the ADS stream to subscribe (as it's not in cache).
- The ADS stream sends a redundant request (same resources, version,
nonce) which the management server ignores.
- The new resource watch sees a "resource-not-found" error once the
watch timer fires.
#### The original fix:
Delay the resource's removal from the cache until the unsubscribe
request was transmitted over the wire, a change implemented in
https://github.com/grpc/grpc-go/pull/8369. However, this solution
introduced new complications:
- The resource's removal from the xdsClient's cache became an
asynchronous operation, occurring while the unsubscribe request was
being sent.
- This asynchronous behavior meant the state maintained within the ADS
stream could still diverge from the cache's state.
- A critical section was absent between the ADS stream's message
transmission logic and the xdsClient's cache access, which is performed
during subscription/unsubscription by its users.
#### The root cause of the previous seen races can be put down two
things:
- Batching of writes for subscribe and unsubscribe calls
- After batching, it may appear that nothing has changed in the list of
subscribed resources, even though a resource was removed and added
again, and therefore the management server would not send any response.
It is important that the management server see the exact sequence of
subscribe and unsubscribe calls.
- State maintained in the ADS stream going out of sync with the state
maintained in the resource cache
#### How does this PR address the above issue?
This PR simplifies the implementation of the ADS stream by removing two
pieces of functionality
- Stop batching of writes on the ADS stream
- If the user registers multiple watches, e.g. resource `A`, `B`, and
`C`, the stream would now send three requests: `[A]`, `[A B]`, `[A B
C]`.
- Queue the exact request to be sent out based on the current state
- As part of handling a subscribe/unsubscribe request, the ADS stream
implementation will queue the exact request to be sent out. When
asynchronously sending the request out, it will not use the current
state, but instead just write the queued request on the wire.
- Don't buffer writes when waiting for flow control
- Flow control is already blocking reads from the stream. Blocking
writes as well during this period might provide some additional flow
control, but not much, and removing this logic simplifies the stream
implementation quite a bit.
RELEASE NOTES:
- xdsclient: fix a race in the xdsClient that could lead to
resource-not-found errors