Context
Legacy pilots: Pilots sent by DIRAC with a proxy embedding a DiracX user token
DiracX pilots: Pilots sent by DiracX with a secret to exchange for a DiracX pilot token
DIRAC Migration
We have to migrate some functionalities from DIRAC to DiracX:
Functions not developped
These functions are only used in DIRAC Web, and won't be migrated to DiracX:
countPilots(countDict, *older, *newer, *timestamp)getPilotStatistics(attribute, selectDict)getPilotMonitorSelectors(selectDict, sortList, startItem, maxItems)[ ] getPilotMonitorWeb(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?)[ ] getPilotSummaryWeb(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?)[ ] getGroupedPilotSummary(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?)[ ] deletePilots
Not distributed functions:
getCurrentPilotCounters(*attrDict)
Function not adapted:
killPilot(pilotRefList), see Moved killPilot to diracAdmin itself DIRAC#8203 getPilotLoggingInfo(pilotReference)storePilotOutPut(pilotReference, output, error)getRemotePilotOutput(pilotReference, pilotDict)getPilotOutput(pilotReference, pilotDict)
Function developped
We kept features from DIRAC, and added a working logging system.
Pilot management
Pilot management functionnalities only reproduces what's being done in DIRAC (pilot creation, deletion, patch, ...). It is mostly done in:
DiracX for the logic, endpoints: feat: add pilot database and router #570
DIRAC for the legacy adaptor (forwarding requests to DiracX): feat: Migrating pilot client toward DiracX DIRAC#8233
Organization
We can decompose all features as such (POST/PATCH/... ... are DiracX endpoints, what's underneath are DIRAC features):
POST /api/pilots/summary
POST /api/pilots/
PATCH /api/pilots/fields
Needs tests and documentation setPilotBenchmarksetAccountingFlagsetPilotStatus
PATCH /api/pilots/jobs
GET /api/pilots/search
Needs documentation getPilotInfo(pilotReference)getPilotsselectPilots
Pilot auth
Pilot auth is pretty straightforward and follows the security model (https://diracx.io/en/latest/admin/reference/security_model/ ): A pilot will exchange a secret (string) for a pilot token.
A pilot token is like a user DiracX token, but without properties (roles), and with a stamp in it. It helps to differenciate a user from a pilot (whereas with legacy pilots, they have the same token as a user except for roles).
It is mostly done in #421 .
Logging system
Logging system is divided in multiple PRs and POCS (proofs of concepts).
DiracX pilots
Current plan is only DiracX pilots can send logs:
They arrive with a secret
They exchange it for a token
They call an endpoint to send logs
They can refresh their pilot tokens on a pilot-dedicated endpoint
Done in:
Pilot side
Pilot auth: feat: Adding JWT support alongside X509 auth Pilot#248
UPDATE(factorized with legacy pilots) Pilot logs: feat: Add pilot pilot logging (legacy and DiracX) Pilot#260
DiracX side
Pilot auth: Adding pilot registrations and authentification (Router) #421
Pilot logs: Add pilot logging #550
Legacy pilots
Update after further discussions, legacy pilots could send logs:
They arrive with a proxy with an user token embedded in it (as currently)
They extract this token
They call an endpoint to send logs
They can refresh their user tokens on a user -dedicated endpoint
Done in:
Pilot logs: feat: Add pilot pilot logging (legacy and DiracX) Pilot#260
DiracX side: [POC] Pilot legacy logging #596
Tested in a DIRAC PR where I create the right environment for legacy pilots to send logs to DiracX: DIRACGrid/DIRAC#8268
New Pilot Features in DiracX
Context
DIRAC Migration
We have to migrate some functionalities from DIRAC to DiracX:
Functions not developped
These functions are only used in DIRAC Web, and won't be migrated to DiracX:
countPilots(countDict, *older, *newer, *timestamp)getPilotStatistics(attribute, selectDict)getPilotMonitorSelectors(selectDict, sortList, startItem, maxItems)[ ](<->getPilotMonitorWeb(selectDict, sortList, startItem, maxItems)getPilotSummary?)[ ](<->getPilotSummaryWeb(selectDict, sortList, startItem, maxItems)getPilotSummary?)[ ](<->getGroupedPilotSummary(selectDict, sortList, startItem, maxItems)getPilotSummary?)[ ]deletePilotsNot distributed functions:
getCurrentPilotCounters(*attrDict)Function not adapted:
killPilot(pilotRefList), see Moved killPilot to diracAdmin itself DIRAC#8203getPilotLoggingInfo(pilotReference)storePilotOutPut(pilotReference, output, error)getRemotePilotOutput(pilotReference, pilotDict)getPilotOutput(pilotReference, pilotDict)Function developped
We kept features from DIRAC, and added a working logging system.
Pilot management
Pilot management functionnalities only reproduces what's being done in DIRAC (pilot creation, deletion, patch, ...). It is mostly done in:
Organization
We can decompose all features as such (
POST/PATCH/... ...are DiracX endpoints, what's underneath are DIRAC features):POST /api/pilots/summarygetPilotSummaryPOST /api/pilots/addPilotReferencesPATCH /api/pilots/fieldssetPilotBenchmarksetAccountingFlagsetPilotStatusPATCH /api/pilots/jobssetJobForPilotGET /api/pilots/searchgetPilotInfo(pilotReference)getPilotsselectPilotsPilot auth
Pilot auth is pretty straightforward and follows the security model (https://diracx.io/en/latest/admin/reference/security_model/): A pilot will exchange a secret (string) for a pilot token.
A pilot token is like a user DiracX token, but without properties (roles), and with a stamp in it. It helps to differenciate a user from a pilot (whereas with legacy pilots, they have the same token as a user except for roles).
It is mostly done in #421.
Logging system
Logging system is divided in multiple PRs and POCS (proofs of concepts).
DiracX pilots
Current plan is only DiracX pilots can send logs:
Done in:
Legacy pilots
Update after further discussions, legacy pilots could send logs:
Done in:
Tested in a DIRAC PR where I create the right environment for legacy pilots to send logs to DiracX: DIRACGrid/DIRAC#8268
New Pilot Features in DiracX
POST /api/auth/secret-exchange)[ ] Owner-Based Restrictions(No owner)