RSDK-9440 Report machine `state` through `GetMachineStatus` #4616

benjirewis · 2024-12-10T20:57:39Z

RSDK-9440

Changes:

Adds a new enumerated State field to robot.MachineStatus both server and client side
Starts machines with a "minimal" config to start web service earlier before starting with full config
Reports StateInitializing in robot.MachineStatus before reconfigure with full config occurs
Reports StateRunning in robot.MachineStatus after reconfigure with full config occurs
Exposes a SetInitializing method on robot.LocalRobot for the above two points to work

Testing:

Modifies server and client side MachineStatus tests to make assertions on State
Basic integration test
Checks for state running before returning from client.New when in testing
Adds more injected machine status functions to client and client session tests

benjirewis

Still a WIP wrt testing; will leave in draft. These are my ideas so far, though.

cheukt · 2024-12-10T21:25:40Z

web/server/entrypoint.go

+	// and immediately start web service. We need the machine to be reachable
+	// through the web service ASAP, even if some resources take a long time to
+	// initially configure.
+	minimalProcessedConfig := &(*fullProcessedConfig)


maybe CopyPublicFields? might look less janky

cheukt · 2024-12-10T21:27:40Z

web/server/entrypoint.go

+	if err := web.RunWeb(ctx, myRobot, options, s.logger); err != nil {
+		return err
+	}
+	myRobot.Reconfigure(ctx, fullProcessedConfig)


is there a risk that the config watcher would call reconfigure before this reconfigure is called?

Yes; good call out. Discussed offline a bit, we should start the config watcher goroutine only after this reconfigure is called.

That does mean that we have the same behavior as today: when the robot is starting up (both in first robotimpl.New(minimalConfig) and myRobot.Reconfigure(fullProcessedConfig)) no new config changes will be seen. So, if a user messes up their config and accidentally starts a module that takes forever to start up, they will not be able to quickly remove that module from their config. Instead, they'll have to restart/shutdown their robot if they want to stop the initial construction. Once again, I don't think this is different from what we have currently, and, of course, viam-server is receptive to gRPC requests earlier with the changes in this PR.

benjirewis · 2024-12-11T21:32:24Z

I broke a lot of tests that I'm presuming are expecting resources to be available as soon as the web service is available. Thinking about it.

dgottlieb · 2024-12-16T14:13:28Z

web/server/entrypoint.go

+	minimalProcessedConfig.Modules = nil
+	minimalProcessedConfig.Processes = nil
+
+	myRobot, err := robotimpl.New(ctx, minimalProcessedConfig, s.logger, robotOptions...)


Is this the best way of achieving what we want or the most expedient?

I'm fine with this as-is. And I'm kind of fine never coming back to think about this. But the whole "robot owns the web server" feels backwards.

There would be less states to consider if we could start a web service and register robots with it. And there'd be a small API that describes:

What state the robot is in (startup or running) and

which APIs are available, e.g:

just "GetMachineStatus" and maybe "ResourceNames"

but none of "SetPower"/other resource specific APIs

But in this PR we have 90-100 lines between this comment/robot.New and when the web service is started. That's a lot of lines to accidentally break our contract and add some blocking code.

Is this the best way of achieving what we want or the most expedient?

Perhaps not, and I understand your argument there. I've introduced a slightly different/simpler mechanic for controlling the "initializing" value, so that might address some of your concerns here. I didn't go so far as starting a web service and registering robots with it (if I'm understanding what you're saying.)

dgottlieb · 2024-12-16T14:14:47Z

web/server/entrypoint.go

+	// Use `fullProcessedConfig` as the initial `oldCfg` for the config watcher
+	// goroutine, as we want incoming config changes to be compared to the full
+	// config.
+	oldCfg := fullProcessedConfig
 	utils.ManagedGo(func() {


It's probably about time this lambda gets its own function/name. I think a lot of my above concern goes away if this 60 lines of control flow keyword soup is hidden by default.

Great idea; working on it + will re-request review when done.

dgottlieb · 2024-12-16T14:17:08Z

web/server/entrypoint.go

@@ -479,7 +502,8 @@ func (s *robotServer) serveWeb(ctx context.Context, cfg *config.Config) (err err
 	}()
 	defer cancel()

-	options, err := s.createWebOptions(processedConfig)
+	// Create initial web options with `minimalProcessedConfig`.
+	options, err := s.createWebOptions(minimalProcessedConfig)


I can't comment off-diff. The goroutine spun off above will check for diff.NetworkEqual and if not, run myRobot.StartWeb (newline 490).

Just below this we call web.RunWeb. I'm not sure what the significance is between having different methods, StartWeb and RunWeb, but assuming that's not interesting: is it possible for those two things to race? Are we guaranteed to end up with the right set of weboptions?

To clarify, this is a question about existing behavior. I don't think this patch changed anything here.

The goroutine spun off above will check for diff.NetworkEqual and if not, run myRobot.StartWeb (newline 490).

Correct; I believe there's a call to StopWeb before that happens, too. And then a Reconfigure after that StopWeb. All about handling network changes in the config.

Just below this we call web.RunWeb. I'm not sure what the significance is between having different methods, StartWeb and RunWeb, but assuming that's not interesting

It's interesting having those two methods. StartWeb starts up the web service on the robot. RunWeb does that, but also waits on <-ctx.Done(), so it's a blocking call and represents the "main" program that "runs" when you call go run web/cmd/server/main.go.

Is it possible for those two things to race? Are we guaranteed to end up with the right set of weboptions?

It depends what you mean by race. There is a lock on starting the web service, so I'm not sure we'd see a race manifest as an actual DATA RACE, but I think you are "right" in wondering about the "right set of weboptions." I'm not entirely sure, but I think RunWeb would run into an error if it tried to start the web service with an old set of options after the config watcher goroutine had started it already with a new set of options. So, my guess is we'd see an error from RunWeb and an inability to start the server in the event of the race you're describing.

dgottlieb · 2024-12-16T14:25:57Z

robot/impl/local_robot.go

@@ -498,6 +502,8 @@ func newWithResources(
 	}

 	successful = true
+	// Robot is "initializing" until first reconfigure after initial creation completes.
+	r.initializing.Store(true)


Technically there's a Reconfigure earlier on newcode 491 that sets this value to false. Not to mention the initializing value is initialized (ugh) to false. Two things:

I'm taking that it's important we exit this function with initializing set to true. But I would expect to see the setting up at the top near the constructor. Can we document that the placement here is intentional to avoid prior calls mucking with the state?

Is this function guaranteed to not start webserver and expose the GetMachineStatus API? If it can, it seems we might be setting initializing too late and may allow clients to observe an illegal transition of ready -> initializing.

Also -- line 428 (old) 432 (new) refers to the mod manager web server. Do we need to consider/provide guidelines for how module SDKs (which are -- in theory -- different from "application SDKs") use and perhaps expose GetMachineStatus?

I've modified the mechanics here slightly. You'll see there's a new robot.Option to start a robot in initializing mode. You can then use SetInitializing(false) to mark the robot as running. This means that only the code here in web/server/entrypoint.go is "special" with respect to initialization. All other calls to robotimpl.New will create robots that always return robot.StateRunning from MachineStatus.

I'm not sure that will address all your concerns here, and I'll think a bit harder about your module question before re-requesting review.

dgottlieb · 2024-12-16T14:33:30Z

robot/impl/local_robot.go

+	// been closed above. This ensures processes are shutdown before any files
+	// are deleted they are using.
+	//
+	// If initializing, machine will be starting with no modules, but may


I'm not sure I understand this. Does anything actually go wrong if we don't guard this "cleanup" logic? Or are we just suggesting that making these calls would be "wasteful" no-ops?

The existing comment/first paragraph refers to "cleanup unused packages", so if we never started up with any, where are they coming from? Existing files on the file system from a previous start?

@cheukt had mentioned it would be good to guard the lines below based on initialization.

Or are we just suggesting that making these calls would be "wasteful" no-ops?

That's my understanding, yep.

Existing files on the file system from a previous start?

Also my understanding, yep.

benjirewis · 2024-12-18T20:57:28Z

robot/client/client_session_test.go

@@ -107,8 +107,11 @@ func TestClientSessionOptions(t *testing.T) {
 								return &dummyEcho{Named: arbName.AsNamed()}, nil
 							},
 							ResourceRPCAPIsFunc: func() []resource.RPCAPI { return nil },
-							LoggerFunc:          func() logging.Logger { return logger },
-							SessMgr:             sessMgr,
+							MachineStatusFunc: func(_ context.Context) (robot.MachineStatus, error) {


I inject an almost identical MachineStatusFunc all over the place in client_session_test.go and client_test.go. client.New now calls MachineStatus when in a testing environment to make sure the robot is in a robot.StateRunning and therefore capable of receiving resource API calls.

I thought about placing this functionality in inject/robot.go, but opted to just put it in every robot inject, since we currently do that for ResourceNamesFunc and ResourceByNameFunc.

benjirewis · 2024-12-18T20:59:14Z

robot/client/client_session_test.go

@@ -129,13 +132,11 @@ func TestClientSessionOptions(t *testing.T) {
 								Disable: true,
 							})))
 						}
-						roboClient, err := client.New(ctx, addr, logger, opts...)


In three of the client session tests, I had to move the instantiation of the robot client to be below the injection of the session manager. client.New now calls MachineStatus in testing environments, which is not exempted from session creation. So, the injected robot will try to Start a session from its session manager, and end up panicking if StartFunc is not defined. I did not leave a comment in-line explaining this.

I'd like to meet offline at some point to walk through TestClientSessionOptions, in particular, and add some documentation to what it's testing.

benjirewis · 2024-12-18T21:00:24Z

robot/client/client_test.go

@@ -859,21 +883,24 @@ func TestClientUnaryDisconnectHandler(t *testing.T) {
 			info *grpc.UnaryServerInfo,
 			handler grpc.UnaryHandler,
 		) (interface{}, error) {
+			// Allow a single GetMachineStatus through; return `io.ErrClosedPipe`
+			// after that.


Left a comment here to better describe what this test is doing. We also have to call handler(ctx, req) below, now, as we do want to invoke the injected MachineStatus to understand that the robot is running.

benjirewis · 2024-12-18T21:00:44Z

robot/client/client_test.go


 	t.Run("unary call to connected remote", func(t *testing.T) {
-		t.Helper()


[fly-by] Seemingly incorrect test helper annotations.

benjirewis · 2024-12-18T21:02:06Z

robot/client/client_test.go

 			},
-			1,
+			2, // once for client.New call and once for MachineStatus call


My comment here and below is hopefully at least somewhat self-explanatory, but the idea here is that this test calls client.New and then rc.MachineStatus to assert on the structure of the machine status. Since client.New now calls MachineStatus in testing environments, the count here and below is increased by a factor of two.

benjirewis · 2024-12-18T21:03:20Z

robot/impl/local_robot.go

+	// been closed above. This ensures processes are shutdown before any files
+	// are deleted they are using.
+	//
+	// If initializing, machine will be starting with no modules, but may


@cheukt had mentioned it would be good to guard the lines below based on initialization.

Or are we just suggesting that making these calls would be "wasteful" no-ops?

That's my understanding, yep.

Existing files on the file system from a previous start?

Also my understanding, yep.

benjirewis · 2024-12-18T21:27:27Z

web/server/entrypoint.go

@@ -479,7 +502,8 @@ func (s *robotServer) serveWeb(ctx context.Context, cfg *config.Config) (err err
 	}()
 	defer cancel()

-	options, err := s.createWebOptions(processedConfig)
+	// Create initial web options with `minimalProcessedConfig`.
+	options, err := s.createWebOptions(minimalProcessedConfig)


The goroutine spun off above will check for diff.NetworkEqual and if not, run myRobot.StartWeb (newline 490).

Correct; I believe there's a call to StopWeb before that happens, too. And then a Reconfigure after that StopWeb. All about handling network changes in the config.

Just below this we call web.RunWeb. I'm not sure what the significance is between having different methods, StartWeb and RunWeb, but assuming that's not interesting

It's interesting having those two methods. StartWeb starts up the web service on the robot. RunWeb does that, but also waits on <-ctx.Done(), so it's a blocking call and represents the "main" program that "runs" when you call go run web/cmd/server/main.go.

Is it possible for those two things to race? Are we guaranteed to end up with the right set of weboptions?

It depends what you mean by race. There is a lock on starting the web service, so I'm not sure we'd see a race manifest as an actual DATA RACE, but I think you are "right" in wondering about the "right set of weboptions." I'm not entirely sure, but I think RunWeb would run into an error if it tried to start the web service with an old set of options after the config watcher goroutine had started it already with a new set of options. So, my guess is we'd see an error from RunWeb and an inability to start the server in the event of the race you're describing.

benjirewis · 2024-12-18T21:27:56Z

web/server/entrypoint.go

+	minimalProcessedConfig.Modules = nil
+	minimalProcessedConfig.Processes = nil
+
+	myRobot, err := robotimpl.New(ctx, minimalProcessedConfig, s.logger, robotOptions...)


Is this the best way of achieving what we want or the most expedient?

Perhaps not, and I understand your argument there. I've introduced a slightly different/simpler mechanic for controlling the "initializing" value, so that might address some of your concerns here. I didn't go so far as starting a web service and registering robots with it (if I'm understanding what you're saying.)

viambot added the safe to test This pull request is marked safe to test from a trusted zone label Dec 10, 2024

benjirewis changed the title ~~RSDK-9440~~ RSDK-9440 Report machine state through GetMachineStatus Dec 10, 2024

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 10, 2024

benjirewis commented Dec 10, 2024

View reviewed changes

benjirewis requested review from dgottlieb and cheukt December 10, 2024 21:14

cheukt reviewed Dec 10, 2024

View reviewed changes

benjirewis mentioned this pull request Dec 11, 2024

DATA-3441 Update data export command #4596

Merged

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 11, 2024

benjirewis force-pushed the machine-state branch from febcd0b to a88c3fa Compare December 11, 2024 19:25

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 11, 2024

benjirewis added 8 commits December 13, 2024 11:38

initial impl

fe92e07

atomic

bc21b4f

start config watcher goroutine last

28b89db

addrderef -> CopyOnlyPublicFields

a634d19

update api dependency and lint

f7dd78c

more lint sigh

48e6714

basic entrypoint test

70a5096

reconfigure in goroutine and check for state running in client new

a4d585d

benjirewis force-pushed the machine-state branch from 533ae80 to a4d585d Compare December 13, 2024 17:28

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 13, 2024

dgottlieb reviewed Dec 16, 2024

View reviewed changes

move out of initializing in setupLocalRobot

eaa1285

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2024

add log lines for debugging 32 bit tests

e6b8b44

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2024

move running check higher

d2dc450

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2024

benjirewis added 2 commits December 18, 2024 11:23

potentially simpler API

b932fdf

more missing injections

aa1020e

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 18, 2024

fix more tests sigh

60ab517

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 18, 2024

maybe this time

4d29ba8

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 18, 2024

actually use value from options

afcfbee

benjirewis commented Dec 18, 2024

View reviewed changes

viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 18, 2024

benjirewis marked this pull request as ready for review December 18, 2024 21:28

benjirewis requested a review from a team as a code owner December 18, 2024 21:28

benjirewis requested review from njooma and purplenicole730 December 18, 2024 21:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RSDK-9440 Report machine `state` through `GetMachineStatus` #4616

RSDK-9440 Report machine `state` through `GetMachineStatus` #4616

benjirewis commented Dec 10, 2024 •

edited

Loading

benjirewis left a comment •

edited

Loading

cheukt Dec 10, 2024

benjirewis Dec 11, 2024

cheukt Dec 10, 2024

benjirewis Dec 11, 2024

benjirewis commented Dec 11, 2024

dgottlieb Dec 16, 2024

benjirewis Dec 18, 2024

dgottlieb Dec 16, 2024

benjirewis Dec 18, 2024

dgottlieb Dec 16, 2024 •

edited

Loading

benjirewis Dec 18, 2024

dgottlieb Dec 16, 2024 •

edited

Loading

dgottlieb Dec 16, 2024

benjirewis Dec 18, 2024

dgottlieb Dec 16, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024

benjirewis Dec 18, 2024


		t.Run("unary call to connected remote", func(t *testing.T) {
		t.Helper()

RSDK-9440 Report machine state through GetMachineStatus #4616

Are you sure you want to change the base?

RSDK-9440 Report machine state through GetMachineStatus #4616

Conversation

benjirewis commented Dec 10, 2024 • edited Loading

benjirewis left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjirewis commented Dec 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgottlieb Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgottlieb Dec 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RSDK-9440 Report machine `state` through `GetMachineStatus` #4616

RSDK-9440 Report machine `state` through `GetMachineStatus` #4616

benjirewis commented Dec 10, 2024 •

edited

Loading

benjirewis left a comment •

edited

Loading

dgottlieb Dec 16, 2024 •

edited

Loading

dgottlieb Dec 16, 2024 •

edited

Loading