-_g͗ddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZmZdd lmZeeeee d ZGd dZy)N)quote)load) InitSystem)JujuSSH)SSHControlPersist)LocalTransport) OCTransport)SaltStackMaster)CommandTimeoutExceptionConnectionExceptionUnsupportedHostExceptionInvalidTransportException)sos_parse_version)localcontrol_persistoc saltstackjujuc`eZdZ d3dZedZdZdZdZedZ dZ d Z d Z d Z d4d Zed ZdZdZdZdZdZdZdZdZdZd4dZdZdZdZdZ d5dZdZdZ dZ!d Z"d!Z#d"Z$d#Z%d$Z&d%Z'd&Z(d'Z)d(Z*d)Z+d*Z,d+Z-d,Z.d-Z/d.Z0d/Z1d0Z2d1Z3d2Z4y)6SosNodeNcF|j|_||_|d|_|j |d|_|d|_|d|_|d|_|jj|_ |jj|_ d|_ d|_ d|_ d|_i|_i|_|xs|jj"|_|jj&s1|jj(s|j$|j_|r||j_d|_d|_d|_g|_dgggg|dd|_d |_t7j8d |_t7j8d |_|j?||_ |j@jBr|j@jBd k7|_ |j@jE|j$|jK|_ |j@j|_|jr|jjLrd}|jNr|r|js|j@jQy|jr!|jSrd|j_*|jjTr|jW|jYy#tF$r}|jId |d}~wwxYw)N cmdlineoptstmpdirhostlen need_sudo sos_optionsFsos_cmd)versionenableddisabledoptionspresetsr sosreportsossos_uirootzUnable to open remote session: )-stripaddresscommonsopts_assign_config_optsrrrrnode_config_fileinherit_config_filerhostclusterhostname sos_env_vars _env_varspassword _password nopasswd_sudosudo_pwsos_path retrievedhash_retrieved file_listsos_infosos_binlogging getLoggersoslogui_log_load_remote_transport _transport default_userconnect Exception log_errordetermine_host_policyno_local connected disconnectcheck_in_container containerizedcreate_sos_container_load_sos_info)selfr(r)r3 local_sudo load_factserrs 7/usr/lib/python3/dist-packages/sos/collector/sosnode.py__init__zSosNode.__init__,s}}  M*    "h' y)  -"=1 $ : :#'99#@#@     !7TYY%7%7yy&&tyy/@/@ $DII   *DII  #y)   # ''. ''1 55g> ?? ' '!__99VCDN  OO # #DNN 3..0 00 ::$)),,J >>j99**,zz**,.3DII+yy&&))+    !+  NN>rWc^|js|jr/tjj |_|jS|j d}|ddk(rB|dj dD]+}|s|j d}|d|j|d<-|jS)Nz env --nullstatusroutput=r`)r2rosenvironcopy run_commandsplit)rOretln_vals rSenv_varszSosNode.env_varss~~zz!#!2~~&&|4x=A%!(m11&9:!$!xx}26q'tAw/ : ~~rWc||_|jjd|j|jjd|jj|jjd|j d|jjdd|jjd|j jy) z>Set the manifest section that this node will write to r0policy sos_versionrfinal_sos_commandrar\N)manifest add_fieldrbr.os_release_namer;rBname)rOrxs rSset_node_manifestzSosNode.set_node_manifests!   DNN; $))*C*CD  t}}Y/GH  3R8  T__-A-ABrWctjjdr|jdytjj d|jdyy)zV Tries to identify if we are currently running in a container or not. z/run/.containerenvz/Found /run/.containerenv. Running in container.T containerz/Found env var 'container'. Running in containerF)rkpathexists log_debugrlgetrVs rSrKzSosNode.check_in_containersK 77>>. / NNL M ::>>+ & 2 NNL MrWc|jjrI|jj|jj|j |jj }|j|d}|ddvr|ddk(r(?P(pass|key|secret|PASS|KEY|SECRET).*?=)(?P.*?\s)z\g****** )resub)rOrdregs rS_sanitize_log_msgzSosNode._sanitize_log_msgsPvvc,c22rWcX|jj|j|y)z:Format a ui message that includes host name and formattingN)r@inforercs rSui_msgzSosNode.ui_msgs s+,rWctjdd}d|jd|d|}|jj |y)z#Used to print and log info messagesr`[:] N)inspectstackrbr?rrOrdcallerlmsgs rSrzSosNode.log_infosF#A&4>>"!F82cU3 rWctjdd}d|jd|d|}|jj |y)z$Used to print and log error messagesr`rrrrN)rrrbr?errorrs rSrFzSosNode.log_error sF#A&4>>"!F82cU3 $rWc|j|}tjdd}d|jd|d|}|jj |y)z$Used to print and log debug messagesr`rrrrN)rrrrbr?debug)rOrdrs rSrzSosNode.log_debugsV$$S)#A&$..!6("SE2 #rWc|jjdtjjddk(}|jr|r|S|j j rdt|S|jrd|S|S)zIf we need to provide a sudo or root password to a command, then here we prefix the command with the correct bits USERr&zsu -c zsudo -S ) r2rrkrlrr* become_rootrr)rOris_roots rS _format_cmdzSosNode._format_cmdsq..$$VRZZ^^F-CDN ::'J 99 E#J<( ( >>cU# # rWcTd}d}|jjZ|jjj|jj}|udj |d}|dr\|d}nV|j |jjdd}|ddk(r&|d jjd \}}}|r=t|jdd k(r|d z } |d |jddz }||jd<|jdr"|jd|jdn<|j|jj k(s|j#dd|_y|j'drd|_|j(d}|j |dd}|ddk(r|j+|d |j'dr|j-y#t$r}|jd |Yd}~d}~wwxYw)zLQueries the node for information about the installed version of sos N.rreleaseT use_containerrrgrrh-z.0z#Unable to fully parse sos release: zsos version is z!sos is not installed on this nodeFz4.0z sos reportz -l3.6)r.container_version_commandpackage_manager pkg_version sos_pkg_namejoinrnr'rolenrErr;rr(r*primaryrFrIcheck_sos_versionr<_load_sos_plugins_load_sos_presets) rOverrelpkgpkgs_rRrsosinfos rSrNzSosNode._load_sos_info#s 99 . . 6))++77 8N8NOChhs9~.y>i.C##DII$G$G26$$HDH~""8n224::3? 3 399S>"a't  L399S>!,-..$' i == # MMODMM),D+EF G<<499#4#44BC"DN  ! !% ('DLc"""3dd"K 8  !  " "78#4 5  ! !% (  " " $1 L!DSEJKK Ls0G?? H'H""H'cL|jd}|j|dd}|ddk(ry|djD]b}|jj ds#|j ddj}|j d j|dyy) Nz --list-presetsTrrgrrhzname:r`r")r<rn splitlinesr' startswithror;append)rOrrlinepnames rSrzSosNode._load_sos_presetsWso.s$$G x=A H 002 ;::<**73 JJw/288:EMM),33E: ; rWcRd}d}d}d}d}|dz|z}|dz|z}|dz|z} |dzdz} |j|||jd<|j|||jd <|j| ||jd <|j| |d |jd <y) Nz,The following plugins are currently enabled:z-The following plugins are currently disabled:z4The following options are available for ALL plugins:z+The following plugin options are available:z Profiles:z(.*?)z rr r!Tprofiles)_regex_sos_helpr;) rOrENABLEDDISABLED ALL_OPTIONSOPTIONSPROFILES enableregdisregoptregproregs rSrzSosNode._load_sos_plugins`s@BL ?g%0 G#k17"X-G#f,#'#7#7 7#K i $($8$8$I j!#'#7#7#H i $($8$8$$O j!rWcg}tj||tjD]j}|jD]U}|s( |j x}r|j |d-|j d} |jd| DWl|S#t $r}|jd|Yd}~~d}~wwxYw)NrzError parsing sos help: ,c3^K|]%}|js|j'ywN)r').0ps rS z*SosNode._regex_sos_help..}sAQqwwyqwwyAs--) rfindallSrrorrErextend) rOregexris_listrresultrlsrRrs rSrzSosNode._regex_sos_helpqsjj6 BF))+ BI!%-2-JJr!u- 3AJJA!AA B B %I)A#'GHHIs&B B> B99B>c |jd||jj|S#t$r!}|j d|d|Yd}~yd}~wwxYw)z1Reads the specified file and returns the contentsz Reading file zException while reading : Nra)rrB read_filerErF)rOto_readrRs rSrzSosNode.read_files\  MMM'3 4??,,W5 5  NN5gYbF G s.1 AAAc |jr:|jd|jdj|jdSt i|j j td|jj|jd}|r!|jd|jd|S|jdt) z[Attempts to identify the host installation against supported distributions zusing local policy ruTz/etc/os-release)cachesysrootinit probe_runtime remote_exec remote_checkzloaded policy z for hostz4Unable to determine host installation. Ignoring node) rrr)rzrr*rrrBrnrrFr )rOr.s rSrGzSosNode.determine_host_policys :: MM%dll8&<&L&L%MN P<<) )"dii&7&7jl"& $ ; ;!%0A!BD  MMN4+?+?*@ J KK MN&&rWc |jd}t|t|k\S#t$r}|jd|Yd}~yd}~wwxYw)aChecks to see if the sos installation on the node is AT LEAST the given ver. This means that if the installed version is greater than ver, this will still return True :param ver: Version number we are trying to verify is installed :type ver: ``str`` :returns: True if installed version is at least ``ver``, else False :rtype: ``bool`` rzError checking sos version: NF)r;rrErF)rOr _node_verrRs rSrzSosNode.check_sos_versionsR  i0I$Y/3DS3II I  NN9#? @ s%( AA  Acj|jsy|jjj|duS)z2Checks if a given package is installed on the nodeFN)r.r pkg_by_name)rOrs rS is_installedzSosNode.is_installeds,yyyy((44S9EErWc(|js]|jsQ|jd |jj |j }|s|jdt  |r1|jjr|jj|}|r|j|}|r|j} | j|}|jj|||||S#t$r}|jd|d}~wwxYw)aRuns a given cmd, either via the SSH session or locally :param cmd: The full command to be run :type cmd: ``str`` :param timeout: Time in seconds to wait for `cmd` to complete :type timeout: ``int`` :param use_shell: If a shell is needed to run `cmd`, set to True :type use_shell: ``bool`` or ``auto`` for transport-determined :param use_container: Run this command in a container *IF* the host is a containerized host :type use_container: ``bool`` :param env: Pass environment variables to set for this `cmd` :type env: ``dict`` z-Node is disconnected, attempting to reconnectzFailed to reconnect to nodez!Error while trying to reconnect: N)rIrrrB reconnectr4r rEr.rLformat_container_commandrrsupdatern) rOrtimeout use_shellrrenv reconnectedrR_cmd_envs rSrnzSosNode.run_commands(~~djj NNJ K "oo77G "NN#@A--# TYY44))44S9C ""3'C }}H//#&C**3C+46 6 !B3%HI s>C.. D7D  Dc& |j}|r|j|n|jd|jr|j |_|jy#t $r}|jd|Yd}~3d}~wwxYw)z.Run an sos report on the node, then collect itz'Unable to determine path of sos archivezError during sos execution: N)execute_sos_commandfinalize_sos_pathrFr7retrieve_sosreportr8rEcleanup)rOrrRs rSr#zSosNode.sosreports A++-D&&t,HI}}!%!8!8!:  A NN9#? @ @ AsAA)) B2B  Bc$||jdvS)z/Verifies if the given preset exists on the noder"r;)rOpresets rS_preset_existszSosNode._preset_existsy111rWcdtfd|jd|jdfDS)z/Verifies if the given plugin exists on the nodec3&K|]}|v ywr)rsplugins rSrz)SosNode._plugin_exists..sE16Q;Err )anyr;rOrs `rS_plugin_existszSosNode._plugin_existss;Ey)A)-z)B)DEE ErWc$||jdvS)z6Checks to see if the plugin is default enabled on noderrrs rS_check_enabledzSosNode._check_enabledrrWc$||jdvS)z7Checks to see if the plugin is default disabled on noder rrs rS_check_disabledzSosNode._check_disabledsz222rWc|jdd}|j|sy|j|r||jjvry|j |r||j dvSy)zAttempts to verify that the given option is available on the node. Note that we only get available options for enabled plugins, so if a plugin has been force-enabled we cannot validate if the plugin option is correct or notrrFr!T)rorr!r*enable_pluginsrr;)rOoptplugs rS_plugin_option_existszSosNode._plugin_option_existssq yy~a ""4(   &DII444   t $$-- 22 2rWc8djfd|DS)z^Returns a comma delimited list for sos plugins that are confirmed to exist on the noderc3FK|]}j|s|ywr)r)rorOs rSrz,SosNode._fmt_sos_opt_list..sBa4+>+>q+ABs!!)r)rOr*s` rS_fmt_sos_opt_listzSosNode._fmt_sos_opt_list sxxB4BBBrWc||_y)zVExpose the node to the cluster profile determined for the environment N)r/)rOr/s rS set_clusterzSosNode.set_clusters  rWc|jjrQ|js|jj|_n)|jd|jjd|jjrE|jjD],}||j vs|j j |.|jjrP|jjD]7|jvs|jj|j<9|jjrs|jjD]Ztfd|jDr"d|jj}|jj |\|jj|r;|jj5|jj|dddy|jj5|jj|dddy#1swYyxYw#1swYyxYw)a+This is used to modify the sos report command run on the nodes. By default, sos report is run without any options, using this will allow the profile to specify what plugins to run or not and what options to use. This will NOT override user supplied options. zCluster specified preset z> but user has also defined a preset. Using user specification.c3&K|]}|v ywrr)rr)r$s rSrz2SosNode.update_cmd_from_cluster..2s;3!8;rrjN)r/ sos_presetrr sos_pluginsr#rrsos_plugin_optionsrplugoptscheck_node_is_primarylockset_primary_optionsset_node_options)rOr%optionr$s @rSupdate_cmd_from_clusterzSosNode.update_cmd_from_clusters << " ";;"ll55  9!%!8!8 9:LLM << # # 00 5t222''..t4 5 << # #||// Jd...,0LL,D,DS,ID$$S) J << * *||66 1;T]];; #uAdll&E&Ec&J%KLFMM((0 1 << - -d 3"" 7 006 7 7"" 4 --d3 4 4 7 7 4 4s$III Iclt|jj|_t|jj|_t|jj|_t|jj |_t|jj |_y)zFrom the global opts configuration, assign those values locally to this node so that they may be acted on individually. N)listr* only_plugins skip_pluginsr#r2rrVs rSr+zSosNode._assign_config_opts>su !!7!78 !7!78"499#;#;<TYY//0 499++, rWc |jd}|j}|r|dt|d}g}|jdrS|jj dk7r:|j dtt|jj |jdrP|jjr:|j dtt|jj|jdr|jjr|j d |jjr|j d |jjr1|j d t|jj|jd r|jjr@|j d tdj|jj|jjr@|j dtdj|jj|jdrP|jjr:|j dtt|jj|jds|jdr|jj dk7r(|j d|jj |jj"r(|j d|jj"|jdre|jj$r(|j d|jj$|jj&r|j d|j)|j+dt,j.j|j0j2|j4}|j6D],}|j6|}|j d|d|r|nd.|j8r|j8Dcgc]J}|j;|j=ddr%|j?|j=ddr|L}}|r3|j d tdjd!|D|j@rb|jC|j@r(|j d"t|j@n|jEd#|j@d$|jFr%|jFDcgc]}|j;|s|}}tI|tI|jFk7r?tKtM|jFtM|z } |jEd%| d&|jO|jF} | r|j d't| |ddj||_(|jSd(|jP|jTjWd)|jPy*|jXr|jXDcgc]}|j[|s|} }tI| tI|jXk7r?tKtM|jXtM| z } |jEd+| d,|jO| } | r|j d-t| |j\r|j\Dcgc]4}||jXvr$|j_|r|j;|r|6}}tI|tI|j\k7r?tKtM|j\tM|z }|jEd.|d/|jO|}|r|j d0t||ddj||_(|jSd(|jP|jTjWd)|jPy*cc}wcc}wcc}wcc}w)1zZUse host facts and compare to the cluster type to modify the sos command if neededr rz --threads=z3.7z--plugin-timeout=z3.8z--allow-system-changesz --no-env-varsz--since=z4.1z--skip-commands=rz --skip-files=z4.2z--cmd-timeout=z4.3z4.2-13r[z--container-runtime=z --namespaces=z4.5.2z--journal-size=z--low-priorityr#z--rarrrjz-k c3 K|]}|ywrr)rr)s rSrz+SosNode.finalize_sos_cmd..s4E1Q4Es z --preset=zRequested to enable preset z" but preset does not exist on nodezRequested plugins z. were requested to be enabled but do not existz--only-plugins=zFinal sos command set to rwNzRequested to skip plugins z%, but plugins are already not enabledz--skip-plugins=zRequested to enable plugins z1, but plugins are already enabled or do not existz--enable-plugins=)0r;determine_sos_labelrrr*threadsrstrplugin_timeoutallow_system_changes no_env_varssince skip_commandsr skip_files cmd_timeoutcontainer_runtime namespaces journal_size low_priorityr8replacerkrr. sos_bin_pathr<rr2rror&rrrr;rr:setr*rrrxryr<rr#r!)rOrlabelsos_optsr$rrr)r*plugsnot_onlyonlyskipnot_skipskiplnnot_onenables rSfinalize_sos_cmdzSosNode.finalize_sos_cmdIsE-- *((*  5<.2G  ! !% (yy  A%*U3tyy7H7H3I-J,K LM  ! !% (yy'''c$))2J2J.K(L'MNP  ! !% (yy-- 89yy$$0yy(5+A*B CD  ! !% (yy&& 2#($))2I2I)J#K"L!NOyy###E#((4993G3G*H$I#JK  ! !% (yy$$$U3tyy/D/D+E%F$GH  ! !% (D,B,B8,Lyy**f4*499+F+F*GHyy###DII$8$8#9:  ! !' *yy%%/$))2H2H1I JKyy%% 01 $$&//  GGLL// >  ## ?C##C(D OObQtt&<= > ? ==#}}E!**1773<?;221773<?CEDE#eCHH4E4E,E&F%G HI ;;""4;;/)E$++,>+? @A!@ @A))$*;*;  ! !% (DD||))#.q12!E7;Eq  rWc2|jj}|r|j|d}|jd}|j d|||_|jdd|_|jjd|j y)z\Use host facts to determine if we need to change the sos path we are retrieving fromrarzFinal sos path: /collected_archiveN) r.sos_path_striprOrorr7archiverxry)rOrpstrips rSr zSosNode.finalize_sos_paths})) <<+Dzz|A (/0 zz#r*   3T\\BrWc||dk(ry|dk(rd|vry|dk(ryt|dkDr|jd ddSd |S) Nrbz+sos report process received SIGKILL on noder`sudozsudo attempt failedz4sos report terminated unexpectedly. Check disk spacer zsos exited with code )rro)rOrcstdouts rSdetermine_sos_errorzSosNode.determine_sos_errorsV 8@ 7, 9I v;?<<%a* *&rd++rWc|jd d}d}d}|jjrd|jj}n|jjr}|jsXd|j j ddd}|jj|jj|d|}nd|jj}|r|jd ||_ |j|j|jjd d d |j }|d d k(r;|djD]s}tj|dr|j!}|j#dr|j dd}O|j#dsa|j d}u|r|j$j'd|t)|dk(r|j$j'dd|St)|dk(r|j$j'dd|S|j$j'dd|S|j$j'dd|S|j+|d |d}|j-d|d d|dt/|#t0$r|j3dt.$r.}|j5d||jd |d!}~wwxYw)"z2Run sos report and capture the resulting file pathzGenerating sos report...Fraz--config-file=z/tmp/rarbz.confr>T)rrrrrrgrrhz*sosreport-*tar*)z sha256 z md5  r`zThe checksum is: rchecksum checksum_typemd5@sha256unknownzError running sos report. rc = z msg = zTimeout exceededz'Exception during sos report execution: zError running sos report: N)rr*r,r-rrrorBcopy_file_to_remote config_filerrnrr1rfnmatchr'rrxryrrmrrEr rFr)rOrrpconfig_file_arg remote_configrrrRs rSr zSosNode.execute_sos_commands ./7 DH Oyy))$24993M3M2N"O..zz&+DKK,=,=c,B2,F+Gu$MMOO77 --%')7}o&FO))>)>(?@$"&,,q0AB ""4<<+/99+<+<-1-115'+'8'8 #:C 8}!M4463Dt-?@#zz|'>?#'::d#3A#6)<=#'::<? 3MM++JA8}* //GKX", //JK //KK MM++OYG K ..s8}c(mL@"%h-H  HIn$&  NN- .   MMCC5I J KK4SE: ;  s8FJ70AJ7+J7;J7J77AJ77#L)LLc|jdz}tjj||j dd} |j |r3|j d|d||jj||S|jd|dy#t$r!}|jd|d |Yd }~yd }~wwxYw) z7Copies the specified file from the host to our temp dirrarbzCopying remote z to local zAttempting to copy remote file %, but it does not exist on filesystemFzFailed to retrieve rN) rrkrrrorrrB retrieve_filerrE)rOrdestdirdestrRs rSr~zSosNode.retrieve_file5s++#ww||GTZZ_R%89 % vZyIJ44T4@@ NN ? sABB C&CCcdj|j} t|jddkr|jd|dy|j |r4|j d|d|}|j |d }|d d k(S|jd |dy#t$r!}|jd|d|Yd}~yd}~wwxYw)z|Removes the spciefied file from the host. This should only be used after we have retrieved the file already rararzRefusing to remove path z0: appears to be incorrect and possibly dangerousFzRemoving file zrm -f Trrgrz!Attempting to remove remote file r}zFailed to remove rN)rrorrrrrnrE)rOrrres rS remove_filezSosNode.remove_fileDswwtzz|$ 4::c?#q(!9$@E EF% tf56tfo&&sd&;8})) NN>tfE== >  NN.tfBqc: ; s$2B/AB/B// C8CCc\|js|jjr |j|j|jd|j|jd |j|j}|rI|jd|jj|jjddy |jd y#t $r|j dYywxYw#t $r}|j |Yd}~yd}~wwxYw) z,Collect the sos report archive from the nodezFailed to make archive readableFzRetrieving sos report from zRetrieving sos report...Nz!Successfully collected sos reportrarbTzFailed to retrieve sos report)rr*rmake_archive_readabler7rErFrr(rr~r:rro)rOrprRs rSrzSosNode.retrieve_sosreportZs >>TYY22 **4==9 3DLL>BC ./ $$T]]3C  KK; < NN ! !$--"5"5c":2"> ? 34 @A   NN3  s)C'/D'DD D+D&&D+c|j |jryd|jvr |jd|jdy|j|j}|s|j dyy)zvRemove the sos report archive from the node, since we have collected it and it would be wasted space otherwiseNr#zNode sos report path z/ looks incorrect. Not attempting to remove pathzFailed to remove sos report)r7rrrrF)rOremoveds rSremove_sos_archivezSosNode.remove_sos_archivepsv == DJJ  dmm + NN24==/BFF G ""4==1 NN8 9rWc|j|jr'dD]"}|j|j|zs"n|jj }|r|j |dyy)z>TYY%:%:!228< ((2NN))(..*=b*AB$$X.NN%=hZ#HI $ %!)A('LM)=hZH4475(:; ! $GsKs## $s@"C%B$A&C%$ C"-+CC%C""C%% D.D  Dcd|}|j|dd}|ddk(ryd}|j|||dfzt)zUsed to make the given archive world-readable, which is slightly better than changing the ownership outright. This is only used when we're not connecting as root. z chmod o+r T)rrrgrz6Exception while making %s readable. Return code was %s)rnrFrE)rOfilepathrrrds rSrzSosNode.make_archive_readablesZ 8*%sB$? x=A F shH 667rW)NNT)F)r[FFN)5__name__ __module__ __qualname__rTpropertyrIrJrArersr|rKrMrrrbrrrrFrrrNrrrrrGrrrnr#rrrr!r&r*r,r8r+r\rAr rmr r~rrrrrrrrWrSrr*sDCG CJ % 8"?  C &P   3 -   2h;P" '$$F IN-1&6P 2E 23"C  &4P -BCH!( C ,:x ,, : 6$, rWr)ryrr=rkrshlexr sos.policiesrsos.policies.init_systemsrsos.collector.transports.jujur(sos.collector.transports.control_persistrsos.collector.transports.localrsos.collector.transports.ocr "sos.collector.transports.saltstackr sos.collector.exceptionsr r r r sos.utilitiesrr]rrrWrSrsc 01F93>AA,(    BBrW