tag:blogger.com,1999:blog-36628948563701669292024-03-14T18:06:26.938+05:30Supreme DebuggingKiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-3662894856370166929.post-4051076585389352302018-06-21T12:32:00.001+05:302018-06-21T12:32:37.344+05:30Numpy array append and performance issues<h3>
<span style="color: blue;">Issue</span></h3>
<div>
Assume you need to add 100 elements in the <a href="http://www.numpy.org/" target="_blank">numpy</a> array, the initial intuition is to run-down the loop construct and append the element to the array.</div>
<div>
<br /></div>
<div>
However, this would be a grave mistake in-terms of extension-ability of the code. Assume that the elements changes from 100 to 50000 - what would you expect? Let's see with below example</div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
<span style="color: blue;">Comparing numpy append vs list append</span> </h3>
<div>
Check out this example (<a href="https://github.com/kumar-m-kiran/techunravel/blob/master/examples/python/numpy_append_performance.py" target="_blank">link</a>) which run's through adding elements </div>
<div>
<ol>
<li>directly to numpy array - using the <a href="https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.append.html" target="_blank">append</a> api provided by numpy</li>
<li>indirectly create the numpy array - using a list (to construct the information) and converting the list to array using numpy <a href="https://docs.scipy.org/doc/numpy-1.14.0/glossary.html#term-array" target="_blank">array</a> api.</li>
</ol>
<div>
We see that the Option(2) is much more efficient compared to Option(1). And more the number of elements to add to numpy array, more efficient is the option (2).</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitCMaCL0GLHmD1CkDfVkB30fsvpRYWkcq2aKhkoma0Hw-xATo5rJOC57OmVvELSqU_FDf-Fdwl9mqzVc4Tuo_jEp30LZL1yb7TEn5F6xgVG9epYSuaHwPMSabxGDRTTduYHf-YSHLIVRG2/s1600/Numpy+array+insert.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="683" data-original-width="1213" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitCMaCL0GLHmD1CkDfVkB30fsvpRYWkcq2aKhkoma0Hw-xATo5rJOC57OmVvELSqU_FDf-Fdwl9mqzVc4Tuo_jEp30LZL1yb7TEn5F6xgVG9epYSuaHwPMSabxGDRTTduYHf-YSHLIVRG2/s320/Numpy+array+insert.gif" width="320" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
<span style="color: blue;">Result Analysis</span></h3>
<div>
Let's look at the sample results obtained in the run</div>
<div>
<br /></div>
<div>
<!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="false"
DefSemiHidden="false" DefQFormat="false" DefPriority="99"
LatentStyleCount="375">
<w:LsdException Locked="false" Priority="0" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index 9"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" Name="toc 9"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="header"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footer"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="index heading"/>
<w:LsdException Locked="false" Priority="35" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of figures"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="envelope return"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="footnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="line number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="page number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote reference"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="endnote text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="table of authorities"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="macro"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="toa heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Bullet 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Number 5"/>
<w:LsdException Locked="false" Priority="10" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Closing"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Signature"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="true"
UnhideWhenUsed="true" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="List Continue 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Message Header"/>
<w:LsdException Locked="false" Priority="11" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Salutation"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Date"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text First Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Note Heading"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Body Text Indent 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Block Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="FollowedHyperlink"/>
<w:LsdException Locked="false" Priority="22" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Document Map"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Plain Text"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="E-mail Signature"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Top of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Bottom of Form"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal (Web)"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Acronym"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Address"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Cite"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Code"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Definition"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Keyboard"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Preformatted"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Sample"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Typewriter"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="HTML Variable"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Normal Table"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="annotation subject"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="No List"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Outline List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Simple 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Classic 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Colorful 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Columns 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Grid 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 4"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 5"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 7"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table List 8"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table 3D effects 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Contemporary"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Elegant"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Professional"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Subtle 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 1"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 2"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Web 3"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Balloon Text"/>
<w:LsdException Locked="false" Priority="39" Name="Table Grid"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Table Theme"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" SemiHidden="true" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" QFormat="true"
Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" QFormat="true"
Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" QFormat="true"
Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" QFormat="true"
Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" QFormat="true"
Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" QFormat="true"
Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" SemiHidden="true"
UnhideWhenUsed="true" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" SemiHidden="true"
UnhideWhenUsed="true" QFormat="true" Name="TOC Heading"/>
<w:LsdException Locked="false" Priority="41" Name="Plain Table 1"/>
<w:LsdException Locked="false" Priority="42" Name="Plain Table 2"/>
<w:LsdException Locked="false" Priority="43" Name="Plain Table 3"/>
<w:LsdException Locked="false" Priority="44" Name="Plain Table 4"/>
<w:LsdException Locked="false" Priority="45" Name="Plain Table 5"/>
<w:LsdException Locked="false" Priority="40" Name="Grid Table Light"/>
<w:LsdException Locked="false" Priority="46" Name="Grid Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="Grid Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="Grid Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="Grid Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="Grid Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="Grid Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="Grid Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="Grid Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="Grid Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="Grid Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="46" Name="List Table 1 Light"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark"/>
<w:LsdException Locked="false" Priority="51" Name="List Table 6 Colorful"/>
<w:LsdException Locked="false" Priority="52" Name="List Table 7 Colorful"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 1"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 1"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 1"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 1"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 1"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 1"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 2"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 2"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 2"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 2"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 2"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 2"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 3"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 3"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 3"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 3"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 3"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 3"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 4"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 4"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 4"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 4"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 4"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 4"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 5"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 5"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 5"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 5"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 5"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 5"/>
<w:LsdException Locked="false" Priority="46"
Name="List Table 1 Light Accent 6"/>
<w:LsdException Locked="false" Priority="47" Name="List Table 2 Accent 6"/>
<w:LsdException Locked="false" Priority="48" Name="List Table 3 Accent 6"/>
<w:LsdException Locked="false" Priority="49" Name="List Table 4 Accent 6"/>
<w:LsdException Locked="false" Priority="50" Name="List Table 5 Dark Accent 6"/>
<w:LsdException Locked="false" Priority="51"
Name="List Table 6 Colorful Accent 6"/>
<w:LsdException Locked="false" Priority="52"
Name="List Table 7 Colorful Accent 6"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Mention"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Smart Hyperlink"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Hashtag"/>
<w:LsdException Locked="false" SemiHidden="true" UnhideWhenUsed="true"
Name="Unresolved Mention"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;
mso-font-charset:0;
mso-generic-font-family:swiss;
mso-font-pitch:variable;
mso-font-signature:-536859905 -1073732485 9 0 511 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:Calibri;
mso-fareast-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
table.MsoTable15Grid1LightAccent1
{mso-style-name:"Grid Table 1 Light - Accent 1";
mso-tstyle-rowband-size:1;
mso-tstyle-colband-size:1;
mso-style-priority:46;
mso-style-unhide:no;
border:solid #B4C6E7 1.0pt;
mso-border-themecolor:accent1;
mso-border-themetint:102;
mso-border-alt:solid #B4C6E7 .5pt;
mso-border-themecolor:accent1;
mso-border-themetint:102;
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-border-insideh:.5pt solid #B4C6E7;
mso-border-insideh-themecolor:accent1;
mso-border-insideh-themetint:102;
mso-border-insidev:.5pt solid #B4C6E7;
mso-border-insidev-themecolor:accent1;
mso-border-insidev-themetint:102;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Calibri",sans-serif;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
table.MsoTable15Grid1LightAccent1FirstRow
{mso-style-name:"Grid Table 1 Light - Accent 1";
mso-table-condition:first-row;
mso-style-priority:46;
mso-style-unhide:no;
mso-tstyle-border-bottom:1.5pt solid #8EAADB;
mso-tstyle-border-bottom-themecolor:accent1;
mso-tstyle-border-bottom-themetint:153;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTable15Grid1LightAccent1LastRow
{mso-style-name:"Grid Table 1 Light - Accent 1";
mso-table-condition:last-row;
mso-style-priority:46;
mso-style-unhide:no;
mso-tstyle-border-top:.75pt double #8EAADB;
mso-tstyle-border-top-themecolor:accent1;
mso-tstyle-border-top-themetint:153;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTable15Grid1LightAccent1FirstCol
{mso-style-name:"Grid Table 1 Light - Accent 1";
mso-table-condition:first-column;
mso-style-priority:46;
mso-style-unhide:no;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
table.MsoTable15Grid1LightAccent1LastCol
{mso-style-name:"Grid Table 1 Light - Accent 1";
mso-table-condition:last-column;
mso-style-priority:46;
mso-style-unhide:no;
mso-ansi-font-weight:bold;
mso-bidi-font-weight:bold;}
</style>
<![endif]-->
<!--StartFragment-->
<br />
<table border="1" cellpadding="0" cellspacing="0" class="MsoTable15Grid1LightAccent1" style="border-collapse: collapse; border: none; mso-border-alt: solid #B4C6E7 .5pt; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-yfti-tbllook: 1184;">
<tbody>
<tr style="height: .3in; mso-yfti-firstrow: yes; mso-yfti-irow: -1; mso-yfti-lastfirstrow: yes;">
<td style="border-bottom: solid #8EAADB 1.5pt; border: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-alt: solid #8EAADB 1.5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 153; mso-border-bottom-themetint: 153; mso-border-themecolor: accent1; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" valign="top" width="156"><div class="MsoNormal">
Num of elements<o:p></o:p></div>
</td>
<td style="border-bottom: solid #8EAADB 1.5pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-alt: solid #8EAADB 1.5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 153; mso-border-bottom-themetint: 153; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" valign="top" width="156"><div class="MsoNormal">
Time taken (in sec) to directly
add in numpy (A)<o:p></o:p></div>
</td>
<td style="border-bottom: solid #8EAADB 1.5pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-alt: solid #8EAADB 1.5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 153; mso-border-bottom-themetint: 153; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" valign="top" width="156"><div class="MsoNormal">
Time taken (in sec) to add to list and then convert to numpy (B)<o:p></o:p></div>
</td>
<td style="border-bottom: solid #8EAADB 1.5pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-alt: solid #8EAADB 1.5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 153; mso-border-bottom-themetint: 153; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" valign="top" width="156"><div class="MsoNormal">
Approach A is slower compared to B by <o:p></o:p></div>
</td>
</tr>
<tr style="height: .3in; mso-yfti-irow: 0;">
<td style="border-top: none; border: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-themecolor: accent1; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="mso-yfti-cnfc: 4; text-align: center;">
10000<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.112673044205<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.00116991996765<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
96 times<o:p></o:p></div>
</td>
</tr>
<tr style="height: .3in; mso-yfti-irow: 1;">
<td style="border-top: none; border: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-themecolor: accent1; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="mso-yfti-cnfc: 4; text-align: center;">
20000<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.218866825104<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.00390100479126<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
56 times<o:p></o:p></div>
</td>
</tr>
<tr style="height: .3in; mso-yfti-irow: 2;">
<td style="border-top: none; border: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-themecolor: accent1; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="mso-yfti-cnfc: 4; text-align: center;">
30000<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.721004962921<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.00337600708008<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
213 times<o:p></o:p></div>
</td>
</tr>
<tr style="height: .3in; mso-yfti-irow: 3;">
<td style="border-top: none; border: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-themecolor: accent1; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="mso-yfti-cnfc: 4; text-align: center;">
40000<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
1.38639616966<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.00489687919617<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
283 times<o:p></o:p></div>
</td>
</tr>
<tr style="height: .3in; mso-yfti-irow: 4; mso-yfti-lastrow: yes;">
<td style="border-top: none; border: solid #B4C6E7 1.0pt; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-themecolor: accent1; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="mso-yfti-cnfc: 4; text-align: center;">
50000<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.85pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
2.46702504158<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
0.00590395927429<o:p></o:p></div>
</td>
<td style="border-bottom: solid #B4C6E7 1.0pt; border-left: none; border-right: solid #B4C6E7 1.0pt; border-top: none; height: .3in; mso-border-alt: solid #B4C6E7 .5pt; mso-border-bottom-themecolor: accent1; mso-border-bottom-themetint: 102; mso-border-left-alt: solid #B4C6E7 .5pt; mso-border-left-themecolor: accent1; mso-border-left-themetint: 102; mso-border-right-themecolor: accent1; mso-border-right-themetint: 102; mso-border-themecolor: accent1; mso-border-themetint: 102; mso-border-top-alt: solid #B4C6E7 .5pt; mso-border-top-themecolor: accent1; mso-border-top-themetint: 102; padding: 0in 5.4pt 0in 5.4pt; width: 116.9pt;" width="156"><div align="center" class="MsoNormal" style="text-align: center;">
417 times<o:p></o:p></div>
</td>
</tr>
</tbody></table>
<br />
<!--EndFragment--></div>
<div>
So we see that the as the number of elements to insert increases, the factor by which the insert takes time increases drastically (as depicted by below graph)</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<script src="//www.powr.io/powr.js?external-type=blogger"></script>
<br />
<div class="powr-graph" id="aa2bd4fb_1529563940">
</div>
<h3>
<span style="color: blue;">Reason</span></h3>
</div>
<div>
Now, lets look at the api details of <a href="https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.append.html" target="_blank">numpy append</a>. When we look into the details - we get a small note - </div>
<div>
<br /></div>
<blockquote class="tr_bq">
<span style="color: #c27ba0;"><span style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px;">Note that </span><a class="reference internal" href="https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.append.html#numpy.append" style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px; text-decoration-line: none;" title="numpy.append"><code class="xref py py-obj docutils literal" style="background-color: transparent; border-radius: 3px; border: 1px solid rgb(225, 225, 232); font-family: Monaco, Menlo, Consolas, "Courier New", monospace; font-size: 11px; font-weight: bold; padding: 2px 4px; white-space: nowrap;"><span class="pre" style="hyphens: none;">append</span></code></a><span style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px;"> does not occur in-place: a new array is allocated and filled. If </span><em class="xref py py-obj" style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px;">axis</em><span style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px;"> is None, </span><em class="xref py py-obj" style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px;">out</em><span style="background-color: white; font-family: "Open Sans", sans-serif; font-size: 13px;"> is a flattened array.</span></span></blockquote>
<div>
So, unlike the python list, the numpy array is not a linked list implementation & every time we add a element, it copies all existing contents of the array with an additional element space - before inserting. Which is a very very costly operation.</div>
<div>
<br /></div>
<div>
So when you need to build an numpy array, always build it first with python list and then convert it to numpy array.</div>
<div>
<br /></div>
<div>
<br /></div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-23562936680708918732018-05-07T10:45:00.000+05:302018-05-07T15:56:25.853+05:30Use of __repr__ in python<div style="height: 0px;">
<span style="color: blue;"><br /></span></div>
<h2 style="height: 0px;">
<span style="color: blue;">Problem Statement</span></h2>
<div>
<br /></div>
<div>
How many times you are in the process of debugging/ printing/logging your python objects for a critical issue analysis, and you find output which is very obscure??? Quiet Often?<br />
<br /></div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #ce5c00; font-weight: bold;"><</span><span style="color: black;">__main__</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">Point</span> <span style="color: black;">instance</span> <span style="color: black;">at</span> <span style="color: #0000cf; font-weight: bold;">0x7f23e62995f0</span><span style="color: #ce5c00; font-weight: bold;">></span>
</pre>
</div>
<br /></div>
<div>
<h2>
<span style="color: blue;">Actual Reason</span></h2>
</div>
<div>
Consider a 'Point' class which takes in two co-ordinates. Now, when you print the object details, python interpreter understands it as a command to provide the instance details of the object along with its associated class.</div>
<div>
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">class</span> <span style="color: black;">Point</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">__init__</span><span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">x_cord</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">y_cord</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #3465a4;">self</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">x</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">x_cord</span>
<span style="color: #3465a4;">self</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">y</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">y_cord</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">__name__</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #4e9a06;">"__main__"</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">d</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">Point</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">4</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87;">print</span> <span style="color: black;">d</span>
</pre>
</div>
<div>
<br />
So, when you run a simple code provided above, you will see that the output is very complicate details of the class object - which provides very little information.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #ce5c00; font-weight: bold;"><</span><span style="color: black;">__main__</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">Point</span> <span style="color: black;">instance</span> <span style="color: black;">at</span> <span style="color: #0000cf; font-weight: bold;">0x7f23e62995f0</span><span style="color: #ce5c00; font-weight: bold;">></span>
</pre>
</div>
<br />
For some, this might be sufficient. However, for many of us, we need to understand more about the object than just the plain instance details associated with the class.<br />
<br />
<h2>
<span style="color: blue;">Workaround</span></h2>
</div>
<div>
Python has provided a beautiful mechanism to have a workaround with this. Here we use a built-in class member called __repr__.</div>
<div>
<br /></div>
<div>
Now, consider the below modified version of the same class definition.</div>
<div>
<br /></div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">class</span> <span style="color: black;">Point</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">__init__</span><span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">x_cord</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">y_cord</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #3465a4;">self</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">x</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">x_cord</span>
<span style="color: #3465a4;">self</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">y</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">y_cord</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">__repr__</span><span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">return</span> <span style="color: #4e9a06;">'Point in co-ordindate form where x = %s, y = %s'</span> <span style="color: #ce5c00; font-weight: bold;">%</span> <span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">x</span><span style="color: black; font-weight: bold;">,</span><span style="color: #3465a4;">self</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">y</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">__name__</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #4e9a06;">"__main__"</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">d</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">Point</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">4</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87;">print</span> <span style="color: black;">d</span>
</pre>
</div>
<br /></div>
<div>
when you run the above program, the out changes to the text description as shown below<br />
<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">lvtb9</span><span style="color: #ce5c00; font-weight: bold;">:~/</span><span style="color: black;">workspace</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">blog_examples</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">repr_usage</span><span style="color: #a40000;">$</span> <span style="color: black;">python</span> <span style="color: black;">class_repr_usage</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">py</span>
<span style="color: black;">Point</span> <span style="color: black;">in</span> <span style="color: black;">co</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">ordindate</span> <span style="color: black;">form</span> <span style="color: black;">where</span> <span style="color: black;">x</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">y</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">4</span>
</pre>
</div>
<br />
Wonderful!!!<br />
<br />
<h2>
<span style="color: blue;">Conclusion</span></h2>
We might think that these minor tweaks carry little value in overall day-day product development/maintainance.<br />
<br />
However, think again - let's say you need to log the object details when an exception is thrown for a code which is legacy and difficult to change. Now just by extending the details of the '__repr__' in your legacy code, you will be easily able to extend the logging details and equip yourself for effective debugging.<br />
<br />
<br /></div>
<div>
<br /></div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-68794379861326311382018-04-27T22:16:00.000+05:302018-04-27T22:23:44.199+05:30Python name guard and its importance<h3>
</h3>
<h2>
<span style="color: blue;">Why do we need "__main__" guard in python code?</span></h2>
We all have seen code where we have protected the python code with "__main__" guard. Why do we need this? take a look at below explanation<br />
<br />
<h3>
File - grepinfiles.py</h3>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">sys</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">grep</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">with</span> <span style="color: #204a87;">open</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">)</span> <span style="color: #204a87; font-weight: bold;">as</span> <span style="color: black;">f</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">line</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">f</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">ptrn</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">line</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">yield</span> <span style="color: black;">line</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">rstrip</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">'\n'</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">],</span><span style="color: black;">sys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">]</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">matchline</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">grep</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">print</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">matchline</span><span style="color: black; font-weight: bold;">)</span>
</pre>
</div>
<br />
<br />
For a sample input file<br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">>>cat /tmp/1.txt
This is a sample code for grep
we do no have any example for egrep
</pre>
</div>
<br />
<br />
We get the below output<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">python</span> <span style="color: black;">grepinfiles</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">py</span> <span style="color: black;">egrep</span> <span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">tmp</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">1.</span><span style="color: black;">txt</span>
<span style="color: black;">we</span> <span style="color: #204a87; font-weight: bold;">do</span> <span style="color: black;">no</span> <span style="color: black;">have</span> <span style="color: black;">any</span> <span style="color: black;">example</span> <span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">egrep</span>
</pre>
</div>
<br />
<br />
Now, lets use this module as package in another module.<br />
<br />
<h3>
File: finderror.py</h3>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">sys</span>
<span style="color: #204a87; font-weight: bold;">from</span> <span style="color: black;">grepinfiles</span> <span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">findpattern</span>
<span style="color: black;">txtfl</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">]</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">line</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">findpattern</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">'ERROR'</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">print</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">line</span><span style="color: black; font-weight: bold;">)</span>
</pre>
</div>
<br />
when you run this function, we get the below output.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #ce5c00; font-weight: bold;">>></span> <span style="color: black;">python</span> <span style="color: black;">finderror</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">py</span> <span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">tmp</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">1.</span><span style="color: black;">txt</span>
<span style="color: black;">Traceback</span> <span style="color: black; font-weight: bold;">(</span><span style="color: black;">most</span> <span style="color: black;">recent</span> <span style="color: black;">call</span> <span style="color: black;">last</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">:</span>
<span style="color: black;">File</span> <span style="color: #4e9a06;">"finderror.py"</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">line</span> <span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">in</span> <span style="color: #ce5c00; font-weight: bold;"><</span><span style="color: black;">module</span><span style="color: #ce5c00; font-weight: bold;">></span>
<span style="color: black;">from</span> <span style="color: black;">grepinfiles</span> <span style="color: black;">import</span> <span style="color: black;">findpattern</span>
<span style="color: black;">File</span> <span style="color: #4e9a06;">"/home/user/workspace/blog_examples/python_name_gaurd/grepinfiles.py"</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">line</span> <span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">in</span> <span style="color: #ce5c00; font-weight: bold;"><</span><span style="color: black;">module</span><span style="color: #ce5c00; font-weight: bold;">></span>
<span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sys</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">],</span><span style="color: black;">sys</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">]</span>
<span style="color: black;">IndexError</span><span style="color: #ce5c00; font-weight: bold;">:</span> <span style="color: black;">list</span> <span style="color: black;">index</span> <span style="color: black;">out</span> <span style="color: black;">of</span> <span style="color: black;">range</span>
</pre>
</div>
<br />
Why is this error???<br />
<br />
<h2>
<span style="color: blue;">Magic variable called "__name__"</span></h2>
<div>
Now, lets modify the code slightly and run the same command.</div>
<div>
<br /></div>
<h3>
File: grepinfiles.py</h3>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">sys</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">findpattern</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">print</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Inside the module"</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">__name__</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">with</span> <span style="color: #204a87;">open</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">)</span> <span style="color: #204a87; font-weight: bold;">as</span> <span style="color: black;">f</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">line</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">f</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">ptrn</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">line</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">yield</span> <span style="color: black;">line</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">rstrip</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">'\n'</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">__name__</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #4e9a06;">"__main__"</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">],</span><span style="color: black;">sys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">]</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">matchline</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">findpattern</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">ptrn</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">print</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">matchline</span><span style="color: black; font-weight: bold;">)</span>
</pre>
</div>
<br /></div>
<div>
and<br />
<br />
<h3>
File: finderror.py</h3>
</div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">sys</span>
<span style="color: #204a87; font-weight: bold;">from</span> <span style="color: black;">grepinfiles</span> <span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">findpattern</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">__name__</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #4e9a06;">"__main__"</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">txtfl</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">argv</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">]</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">line</span> <span style="color: #204a87; font-weight: bold;">in</span> <span style="color: black;">findpattern</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">'ERROR'</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">txtfl</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: #204a87; font-weight: bold;">print</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">line</span><span style="color: black; font-weight: bold;">)</span>
</pre>
</div>
<div>
<br /></div>
<br />
Now, check the output for the modified source code<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #ce5c00; font-weight: bold;">>></span> <span style="color: black;">python</span> <span style="color: black;">grepinfiles</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">py</span> <span style="color: black;">egrep</span> <span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">tmp</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">1.</span><span style="color: black;">txt</span>
<span style="color: black; font-weight: bold;">(</span><span style="color: #a40000; font-weight: bold;">'</span><span style="color: black;">Inside</span> <span style="color: black;">the</span> <span style="color: black;">module</span><span style="color: #a40000;">'</span><span style="color: black; font-weight: bold;">,</span> <span style="color: #a40000;">'</span><span style="color: black;">__main__</span><span style="color: #a40000;">'</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">we</span> <span style="color: black;">do</span> <span style="color: black;">no</span> <span style="color: black;">have</span> <span style="color: #204a87; font-weight: bold;">any</span> <span style="color: black;">example</span> <span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black;">egrep</span>
<span style="color: #ce5c00; font-weight: bold;">>></span> <span style="color: black;">python</span> <span style="color: black;">finderror</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">py</span> <span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">tmp</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">1.</span><span style="color: black;">txt</span>
<span style="color: black; font-weight: bold;">(</span><span style="color: #a40000; font-weight: bold;">'</span><span style="color: black;">Inside</span> <span style="color: black;">the</span> <span style="color: black;">module</span><span style="color: #a40000;">'</span><span style="color: black; font-weight: bold;">,</span> <span style="color: #a40000;">'</span><span style="color: black;">grepinfiles</span><span style="color: #a40000;">'</span><span style="color: black; font-weight: bold;">)</span>
</pre>
</div>
<br />
<h2>
<span style="color: blue;">Explanation</span></h2>
<blockquote class="tr_bq">
The main package which is invoked by the python interpreter will have __name__ variable set to __main__</blockquote>
<blockquote class="tr_bq">
Any other module/package which is invoked by main package/module will have __name__ as the module name itself.</blockquote>
<div>
<div>
<br /></div>
<div>
So, when finderror.py was invoked,</div>
</div>
<div>
<ul>
<li>finderror.py module will have __name__ set to __main__</li>
<li>grepinfiles.py module will have __name__ set to 'grepinfiles'</li>
</ul>
<div>
However, when only grepinfiles.py was invoked,</div>
</div>
<div>
<ul>
<li>grepinfiles.py module will have __name__ set to '__main__'</li>
</ul>
<h2>
<span style="color: blue;">Conclusion</span></h2>
</div>
<div>
Name guard is a mechanism to customize your python module/package to run any specific code for the module when invoked independently. Also, it is a mechanism to safeguard the code base which are not to be executed when invoked from other up-stream modules/functions.</div>
<div>
<br /></div>
<div>
<br /></div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com1tag:blogger.com,1999:blog-3662894856370166929.post-58563621130259602342018-03-31T13:21:00.000+05:302018-03-31T21:12:07.717+05:30How to use linux perf tools and save dollars - Part 2<h2>
Flame Graph - Introduction</h2>
<div>
In our <a href="https://techunravel.blogspot.com/2018/03/how-to-use-linux-perf-tool-and-save.html" target="_blank">previous discussion</a> (I strongly recommend to go through the linked blog), we have seen how in real world, linux perf tool would be helpful in writing a very well optimized code. We have seen an example of how we could save loads of CPU cycles in the example linked. </div>
<div>
<br /></div>
<div>
Now we will see how to visualize the output which would help us to find the bottle-neck super fast!!!</div>
<div>
<br /></div>
<h2>
What is flame graph?</h2>
<div>
<a href="https://github.com/brendangregg/FlameGraph" target="_blank">Flame graph</a> is a tool which converts the binary data file from perf tool into a "*.svg" a more comfortable visualized format. In fact, to put it more generically flame graph is a tool which helps to represent different profile data - like perf tool, Dtrace etc in a visual format.</div>
<div>
<br /></div>
<div>
<br /></div>
<h2>
Ok... Tell me how to use it</h2>
<h3>
Step - 1</h3>
<div>
<a href="https://github.com/brendangregg/FlameGraph.git" target="_blank">Install the flame</a> and generate perf data dump. For this, we are using the <a href="https://techunravel.blogspot.com/2018/03/how-to-use-linux-perf-tool-and-save.html" target="_blank">example</a> in previous discussion - and generate the perf data dump. Check the below GIF which provides the steps executed - </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYTZ1NalX3SBZ-oiyN3Ei3BUV2EdVmp0ug7tvEbN4yi4uzWmvM4RNIzXkThAfvVQjMCw3pmqt4LGs3BukXfxi-bS1BXb3geI_oZ2U0CWmUVDgEoxt09UnKaRmmaGA2Cl4ifk9GFXUTLg-v/s1600/flame+graph.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="645" data-original-width="1062" height="388" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYTZ1NalX3SBZ-oiyN3Ei3BUV2EdVmp0ug7tvEbN4yi4uzWmvM4RNIzXkThAfvVQjMCw3pmqt4LGs3BukXfxi-bS1BXb3geI_oZ2U0CWmUVDgEoxt09UnKaRmmaGA2Cl4ifk9GFXUTLg-v/s640/flame+graph.gif" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Install Flame Graph and using perf generate the perf dump</td></tr>
</tbody></table>
<div>
<br /></div>
<h3>
Step - 2</h3>
<div>
Follow the steps shown in the below GIF to convert the dump into a "*.svg" file which can be opened in any browser and analyzed.</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigx6gTfkE55BubibUQWOgaJsZkImLZf_vSq0RUQHlp-CCptuwxwn4v4FnnK1duLPKGGO71faYmTGNH_M4rRwlUMaT-PuzOwtVxh9QsaIBWJ4WG-t6G9BP51Kl4FAkpseOFY7IALbA0QlUY/s1600/flame+graph+part+2.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="645" data-original-width="1062" height="388" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigx6gTfkE55BubibUQWOgaJsZkImLZf_vSq0RUQHlp-CCptuwxwn4v4FnnK1duLPKGGO71faYmTGNH_M4rRwlUMaT-PuzOwtVxh9QsaIBWJ4WG-t6G9BP51Kl4FAkpseOFY7IALbA0QlUY/s640/flame+graph+part+2.gif" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Convert perf data to flame graph</td></tr>
</tbody></table>
<div>
<br /></div>
<h3>
Step - 3</h3>
<div>
Analysis of the graph</div>
<div>
<br /></div>
<div>
<ul>
<li>In the GIF, you can see that the graph generated most of the stack (represents CPU cycles) is related to <b>usleep</b> system call.</li>
<li>Also, the usleep is triggered by <b>consumer</b> call-back function of the consumer thread.</li>
<li>Compare the total area of the <b>consumer</b> and <b>producer</b> we would be clearly identifying the total CPU consumed by consumer is far more than producer.</li>
</ul>
<div>
This immediately helps us concluded that - consumer slow is excessively slow and is due to the function usleep in the consumer call back function.</div>
</div>
<div>
<br /></div>
<div>
<span style="background-color: white; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif; font-size: 13px;">Óyela.... we will get all required answer!!!!</span></div>
<div>
<span style="background-color: white; font-family: "helvetica neue" , "helvetica" , "arial" , sans-serif; font-size: 13px;"><br /></span></div>
<h2>
Conclusion</h2>
<div>
<ul>
<li>Use the flame graph tool very effectively, (even in multi-threaded applications also) to visualize your application bottle-neck and fix them.</li>
<li>Both the long term costs saved, and time to debug will be sufficiently reduced with this great tool.</li>
</ul>
</div>
<div>
Keeping looking into this space, you would get much more information on this tool.</div>
<div>
Would be happy to hear you out and help you... Please leave your opinion in the comment section.<br />
<br />
<h2>
And a closing thought...</h2>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDoBNaZAelqWHVxvO11DL2qdnoLrD-kc47WBgc69Cu0ytKbX2dNBmsLwqjdekyjIn-ASHMQmMa5L7IlpiJnPDFTlwcKQoFbrezYtwqLzM1ipRTMwaTHFczei_9w8vfWaAbRSYsgtMUEUA_/s1600/You+Win+You+Learn+Motivational+Poster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1132" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDoBNaZAelqWHVxvO11DL2qdnoLrD-kc47WBgc69Cu0ytKbX2dNBmsLwqjdekyjIn-ASHMQmMa5L7IlpiJnPDFTlwcKQoFbrezYtwqLzM1ipRTMwaTHFczei_9w8vfWaAbRSYsgtMUEUA_/s400/You+Win+You+Learn+Motivational+Poster.png" width="282" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-42913962662145214382018-03-31T08:48:00.003+05:302018-03-31T08:49:06.253+05:30How to find linux kernel version in your distros<h2>
How do you find the linux kernel version you are running?</h2>
<div>
Run the below command to find out the Linux kernel version.<br />
<br />
<pre style="background: #f1f1f1; color: black;">uname <span style="color: #2060a0;">-</span>r
</pre>
<br />
In my distros, I get the below response<br />
<br />
<pre style="background: #f1f1f1; color: black;">~/ $ uname -r
3.13.0-142-generic
~/ $
</pre>
<br />
<br />
<br />
Here is a short clip on how to run and get the details.<br />
Added bonus, I have given more details on other options for the same command.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgn1HOBk5PkNXPEqiszXZcPz98kaWODzdvhzDLZZkom__ppcQxp5pD77gpkBR-oGlGdJxKzvjYvucR7CEIMjqUgFJFkSlPeULjva5UyP6SUyfvK9cadDWG7VhEC-NJs0Mr4TM5oVPE36w41/s1600/How+to+find+Kernel+version.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="" border="0" data-original-height="645" data-original-width="1062" height="388" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgn1HOBk5PkNXPEqiszXZcPz98kaWODzdvhzDLZZkom__ppcQxp5pD77gpkBR-oGlGdJxKzvjYvucR7CEIMjqUgFJFkSlPeULjva5UyP6SUyfvK9cadDWG7VhEC-NJs0Mr4TM5oVPE36w41/s640/How+to+find+Kernel+version.gif" title="How to find linux kernel version" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">How to find linux kernel version</td></tr>
</tbody></table>
<br />
<br /></div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-43772312090804632232018-03-30T16:06:00.000+05:302018-03-30T23:09:00.472+05:30SBT run/test for every edit automatically<div dir="ltr" style="text-align: left;" trbidi="on">
<h2>
What is it about?</h2>
<div>
Imagine you have to do huge number of smaller edits to your source code in scala and verify the output. Most of the time, in other languages especially, you end up doing lots of manual work that consumes lots of repeated boring time.</div>
<div>
<br /></div>
<div>
So, we end up taking some time to automate them - change -> build -> compile -> run. Which is smartly handled in scala.</div>
<div>
<br /></div>
<h2>
Ok, tell me about the savior</h2>
<div>
~ (Tilde) is our savior. </div>
<div>
<br /></div>
<div>
Run the program with prefix of ~ to ensure it continuously polls for the change and executes it when detected one.</div>
<div>
<br /></div>
<div>
Like :</div>
<div>
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid blue; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">sbt> ~run
</pre>
</div>
<div>
<br /></div>
<div>
<br />
<h2>
Ok, show me a small clip with example</h2>
</div>
<div>
Here you go</div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCrZ-L7tzlDgsF3Kuh0307TmWERbfQsraOaeRC0HPKtVMBZ_ab8CvRVbqO4unGUdaOPmszVs-ZeHnRUzb3-K23zgfPQEGhKDMNYOI9gjpmwvj6aWY6DF-slq13vGYZkaC7_YkJhMtPmr_p/s1600/Run+with+tilde.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="" border="0" data-original-height="813" data-original-width="1424" height="365" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCrZ-L7tzlDgsF3Kuh0307TmWERbfQsraOaeRC0HPKtVMBZ_ab8CvRVbqO4unGUdaOPmszVs-ZeHnRUzb3-K23zgfPQEGhKDMNYOI9gjpmwvj6aWY6DF-slq13vGYZkaC7_YkJhMtPmr_p/s640/Run+with+tilde.gif" title="scala sbt run on the go" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">GIF on how to use tilde and run on-the-go</td></tr>
</tbody></table>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
Similarly, you can do the same for testOnly command to build your test cases.<br />
<br />
Here you go for the test suite example<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_7-RW2fKuEqpcbeEL7g3Ufp1gk-kELUDpz6ZgPY8etffrwuE_mBay_hYXNkruRSWB2ys6bMNPdJz2H0j5epxvIXxmfzaD3Q3UdllpnFSbeZERgbWZ0q7CWw_NePiO3lTIpHUHrFWnxYhM/s1600/Run+with+tilde.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="813" data-original-width="1425" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_7-RW2fKuEqpcbeEL7g3Ufp1gk-kELUDpz6ZgPY8etffrwuE_mBay_hYXNkruRSWB2ys6bMNPdJz2H0j5epxvIXxmfzaD3Q3UdllpnFSbeZERgbWZ0q7CWw_NePiO3lTIpHUHrFWnxYhM/s640/Run+with+tilde.gif" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">GIF on how to use tilde and run test cases on the go</td></tr>
</tbody></table>
<br />
<br /></div>
<div>
Wonderful, ain't it?</div>
<div>
Let me know your thoughts in the comment section.</div>
</div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-52483684247260214152018-03-29T22:11:00.000+05:302018-03-30T09:38:36.521+05:30How to find which flavor and which version of JDK do I have on my Ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
Command to find the file containing the jdk information</h2>
<div>
Command:</div>
<div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">file <span style="color: #aa5500;">`which java javac`</span>
</pre>
</div>
<br /></div>
<div>
Output:<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">~/ $ file <span style="color: #aa5500;">`which java javac`</span>
/usr/bin/java: symbolic link to `/etc/alternatives/java'
/usr/bin/javac: symbolic link to `/etc/alternatives/javac'
</pre>
</div>
<br />
<br />
<h3 style="text-align: left;">
Command to find the jdk version information</h3>
</div>
<div>
Command and Output</div>
<div>
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">~/ $ file /etc/alternatives/java /etc/alternatives/javac
/etc/alternatives/java: symbolic link to `/usr/lib/jvm/java-8-oracle/jre/bin/java'
/etc/alternatives/javac: symbolic link to `/usr/lib/jvm/java-8-oracle/bin/javac'
~/ $
</pre>
</div>
<div>
<br /></div>
<div>
Now, you are having Java/JDK version 8 of oracle.<br />
<br /></div>
</div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0India20.593684 78.962880000000041-8.6044825 37.654286000000042 49.791850499999995 120.27147400000004tag:blogger.com,1999:blog-3662894856370166929.post-68704896233669015992018-03-29T19:17:00.002+05:302018-03-31T21:07:14.055+05:30How to use linux perf tool and save dollars<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
When does this tool help me?</h2>
<div style="text-align: left;">
Ask yourself if you have any of the following problem</div>
<div style="text-align: left;">
</div>
<ol style="text-align: left;">
<li>Does you application built on linux run slow? And want to find out the reason?</li>
<li>Do you want to be cost-effective and don't want to buy any high licensed profiling product?</li>
<li>Do you want a reliable and accurate profiling tool in the linux distros?</li>
</ol>
<div>
If you have a "yes" for most of the above questions, then this post should help you provide a gateway into the "easy world of profilers".</div>
<div>
<br /></div>
<h2 style="text-align: left;">
What does this tool do?</h2>
<div>
<ul style="text-align: left;">
<li>It is a tool available in linux distros (Linux 2.6+) (<a href="https://techunravel.blogspot.com/2018/03/how-to-find-linux-kernel-version-in.html" target="_blank">How to find linux kernel version?</a>) which abstracts CPU hardware differences and provides a simple way to instrument/profile your code. </li>
</ul>
In plain terms, it is a tool which is helpful in finding out any possible bottle-necks in your applications.<br />
<ul style="text-align: left;">
<li>Unlike tools such as purecoverage/purify/quanitfy series, perf does not need your application to be recompile.</li>
</ul>
<div>
In plain terms, this tool does not add any additional code (called instrumenting) overhead and can run on precompiled binary. It derives all required metrics from the linux low level system parameters thus saving your day.</div>
</div>
<div>
<br /></div>
<div>
Another major worry for any instrument code is that, during the process of instrumenting code, it slows down your application thus sometimes takes the real essence of the tool. perf stands out as one of the big guy, who does not add much overhead to your application during profiling.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
OK, Show me some examples (I mean real example)</h2>
<div>
Consider the below consumer producer problem - where</div>
<div>
<ul style="text-align: left;">
<li>Producer puts data to mutex protected buffer every 1 second.</li>
<li>Consumer consumes the data only when it reaches 10000 messages (It does not want to process every message, it would want to take data in batches.</li>
<li>And we have 100 consumer - producer pair running in our application</li>
</ul>
</div>
<div>
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;"> 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187</pre>
</td><td><pre style="line-height: 125%; margin: 0;"><span style="color: #333333;">/</span> <span style="color: #008800; font-weight: bold;">for</span> sleep
<span style="color: #557799;">#include <unistd.h></span>
<span style="color: #888888;">// streams</span>
<span style="color: #557799;">#include <iostream></span>
<span style="color: #557799;">#include <sstream></span>
<span style="color: #888888;">// pthread</span>
<span style="color: #557799;">#include <pthread.h></span>
<span style="color: #888888;">// memcpy</span>
<span style="color: #557799;">#include <string.h></span>
<span style="color: #888888;">// strings</span>
<span style="color: #557799;">#include <string></span>
<span style="color: #557799;">#include <vector></span>
<span style="color: #008800; font-weight: bold;">using</span> std<span style="color: #333333;">::</span>cout;
<span style="color: #008800; font-weight: bold;">using</span> std<span style="color: #333333;">::</span>endl;
<span style="color: #008800; font-weight: bold;">using</span> std<span style="color: #333333;">::</span>string;
<span style="color: #008800; font-weight: bold;">using</span> std<span style="color: #333333;">::</span>vector;
<span style="color: #008800; font-weight: bold;">using</span> std<span style="color: #333333;">::</span>stringstream;
<span style="color: #557799;">#define INIT_COUNT_BUF_SIZE 4096</span>
<span style="color: #557799;">#define ITER 10</span>
<span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">Mutex</span> {
<span style="color: #997700; font-weight: bold;">public:</span>
Mutex() {
pthread_mutex_init(<span style="color: #333333;">&</span>m, <span style="color: #007020;">NULL</span>);
}
<span style="color: #333333;">~</span>Mutex() {
pthread_mutex_destroy(<span style="color: #333333;">&</span>m);
}
<span style="color: #333399; font-weight: bold;">void</span> Lock() {
pthread_mutex_lock(<span style="color: #333333;">&</span>m);
}
<span style="color: #333399; font-weight: bold;">void</span> Unlock() {
pthread_mutex_unlock(<span style="color: #333333;">&</span>m);
}
<span style="color: #333399; font-weight: bold;">bool</span> Trylock() {
<span style="color: #008800; font-weight: bold;">return</span> (pthread_mutex_trylock(<span style="color: #333333;">&</span>m) <span style="color: #333333;">==</span> <span style="color: #0000dd; font-weight: bold;">0</span>);
}
<span style="color: #997700; font-weight: bold;">private:</span>
<span style="color: #333399; font-weight: bold;">pthread_mutex_t</span> m;
};
<span style="color: #008800; font-weight: bold;">class</span> <span style="color: #bb0066; font-weight: bold;">SyncedBuffer</span> {
<span style="color: #997700; font-weight: bold;">public:</span>
SyncedBuffer() {
mBuffer.reserve(INIT_COUNT_BUF_SIZE);
}
<span style="color: #008800; font-weight: bold;">virtual</span> <span style="color: #333333;">~</span>SyncedBuffer() {
}
<span style="color: #333399; font-weight: bold;">size_t</span> Size() {
Lock();
<span style="color: #333399; font-weight: bold;">size_t</span> ret <span style="color: #333333;">=</span> mBuffer.size();
Unlock();
<span style="color: #008800; font-weight: bold;">return</span> ret;
}
<span style="color: #333399; font-weight: bold;">void</span> Write(<span style="color: #008800; font-weight: bold;">const</span> <span style="color: #333399; font-weight: bold;">char</span><span style="color: #333333;">*</span> src, <span style="color: #333399; font-weight: bold;">size_t</span> n) {
Lock();
mBuffer.push_back(string((<span style="color: #333399; font-weight: bold;">char</span><span style="color: #333333;">*</span>)src, n));
Unlock();
}
<span style="color: #333399; font-weight: bold;">size_t</span> Read(<span style="color: #333399; font-weight: bold;">char</span><span style="color: #333333;">*</span> dst, <span style="color: #333399; font-weight: bold;">size_t</span> n) {
Lock();
memcpy((<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span>)dst, mBuffer.back().c_str(), mBuffer.back().size());
<span style="color: #333399; font-weight: bold;">size_t</span> copy_size <span style="color: #333333;">=</span> mBuffer.back().size();
mBuffer.pop_back();
Unlock();
<span style="color: #008800; font-weight: bold;">return</span> copy_size;
}
<span style="color: #333399; font-weight: bold;">void</span> Reset() {
Lock();
mBuffer.clear();
Unlock();
}
<span style="color: #997700; font-weight: bold;">protected:</span>
<span style="color: #333399; font-weight: bold;">void</span> Lock() {
m.Lock();
}
<span style="color: #333399; font-weight: bold;">void</span> Unlock() {
m.Unlock();
}
vector<span style="color: #333333;"><</span>string<span style="color: #333333;">></span> mBuffer;
Mutex m;
};
<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span> <span style="color: #0066bb; font-weight: bold;">producer</span>(<span style="color: #333399; font-weight: bold;">void</span> <span style="color: #333333;">*</span>ptr) {
string prefix(<span style="background-color: #fff0f0;">""</span>);
SyncedBuffer <span style="color: #333333;">*</span>b <span style="color: #333333;">=</span> (SyncedBuffer<span style="color: #333333;">*</span>)ptr;
cout <span style="color: #333333;"><<</span> prefix <span style="color: #333333;"><<</span> <span style="background-color: #fff0f0;">"producer started"</span> <span style="color: #333333;"><<</span> endl;
string data <span style="color: #333333;">=</span> <span style="background-color: #fff0f0;">"simple data string #"</span>;
<span style="color: #008800; font-weight: bold;">do</span>{
<span style="color: #008800; font-weight: bold;">static</span> <span style="color: #333399; font-weight: bold;">int</span> i <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">1</span>;
stringstream ss;
ss <span style="color: #333333;"><<</span> data <span style="color: #333333;"><<</span> i<span style="color: #333333;">++</span>;
string s <span style="color: #333333;">=</span> ss.str();
b<span style="color: #333333;">-></span>Write(s.c_str(), s.size());
cout<span style="color: #333333;"><<</span><span style="background-color: #fff0f0;">" P "</span><span style="color: #333333;"><<</span>b<span style="color: #333333;">-></span>Size()<span style="color: #333333;"><<</span>endl;
usleep(<span style="color: #0000dd; font-weight: bold;">1000000</span>); <span style="color: #888888;">//Producer pumps data every 3 seconds.</span>
//<span style="color: #008800; font-weight: bold;">if</span>(i <span style="color: #333333;">%</span> <span style="color: #0000dd; font-weight: bold;">100</span> <span style="color: #333333;">==</span> <span style="color: #0000dd; font-weight: bold;">0</span>) cout<span style="color: #333333;"><<</span><span style="background-color: #fff0f0;">"Producer count"</span><span style="color: #333333;"><<</span>i<span style="color: #333333;"><<</span>endl;
} <span style="color: #008800; font-weight: bold;">while</span>(<span style="color: #007020;">true</span>);
cout <span style="color: #333333;"><<</span> prefix <span style="color: #333333;"><<</span> <span style="background-color: #fff0f0;">"producer finished"</span> <span style="color: #333333;"><<</span> endl;
<span style="color: #008800; font-weight: bold;">return</span> (<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span>)<span style="color: #0000dd; font-weight: bold;">0</span>;
}
<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span> <span style="color: #0066bb; font-weight: bold;">consumer</span>(<span style="color: #333399; font-weight: bold;">void</span> <span style="color: #333333;">*</span>ptr) {
string prefix(<span style="background-color: #fff0f0;">"</span><span style="background-color: #fff0f0; color: #666666; font-weight: bold;">\t\t\t\t\t\t</span><span style="background-color: #fff0f0;">"</span>);
SyncedBuffer <span style="color: #333333;">*</span>b <span style="color: #333333;">=</span> (SyncedBuffer<span style="color: #333333;">*</span>)ptr;
<span style="color: #333399; font-weight: bold;">char</span> data[INIT_COUNT_BUF_SIZE];
cout <span style="color: #333333;"><<</span> prefix <span style="color: #333333;"><<</span> <span style="background-color: #fff0f0;">"consumer started"</span> <span style="color: #333333;"><<</span> endl;
<span style="color: #333399; font-weight: bold;">size_t</span> read <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">0</span>;
<span style="color: #008800; font-weight: bold;">do</span>{
<span style="color: #008800; font-weight: bold;">static</span> <span style="color: #333399; font-weight: bold;">int</span> i <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">1</span>;
<span style="color: #008800; font-weight: bold;">if</span>(b<span style="color: #333333;">-></span>Size() <span style="color: #333333;"><=</span> <span style="color: #0000dd; font-weight: bold;">10000</span> )
{
usleep(<span style="color: #0000dd; font-weight: bold;">1</span>); <span style="color: #888888;">//Bad code in consumer to sleep very less time</span>
}
<span style="color: #008800; font-weight: bold;">else</span>
{
read <span style="color: #333333;">=</span> b<span style="color: #333333;">-></span>Read(data, INIT_COUNT_BUF_SIZE);
string s;
s.append(data, read);
i<span style="color: #333333;">++</span>;
}
} <span style="color: #008800; font-weight: bold;">while</span>(<span style="color: #007020;">true</span>);
cout <span style="color: #333333;"><<</span> prefix <span style="color: #333333;"><<</span> <span style="background-color: #fff0f0;">"consumer finished"</span> <span style="color: #333333;"><<</span> endl;
<span style="color: #008800; font-weight: bold;">return</span> (<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span>)<span style="color: #0000dd; font-weight: bold;">0</span>;
}
<span style="color: #333399; font-weight: bold;">int</span> <span style="color: #0066bb; font-weight: bold;">main</span>() {
<span style="color: #008800; font-weight: bold;">const</span> <span style="color: #333399; font-weight: bold;">int</span> COUNT <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">100</span>;
SyncedBuffer b[COUNT];
<span style="color: #333399; font-weight: bold;">pthread_t</span> prod_thread[COUNT];
<span style="color: #333399; font-weight: bold;">pthread_t</span> cons_thread[COUNT];
<span style="color: #008800; font-weight: bold;">for</span> (<span style="color: #333399; font-weight: bold;">int</span> i <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">0</span>; i <span style="color: #333333;"><</span> COUNT; i<span style="color: #333333;">++</span>)
pthread_create(<span style="color: #333333;">&</span>prod_thread[i], <span style="color: #007020;">NULL</span>, producer, (<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span>)<span style="color: #333333;">&</span>(b[i]));
sleep(<span style="color: #0000dd; font-weight: bold;">2</span>); <span style="color: #888888;">// seconds</span>
<span style="color: #008800; font-weight: bold;">for</span> (<span style="color: #333399; font-weight: bold;">int</span> i <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">0</span>; i <span style="color: #333333;"><</span> COUNT; i<span style="color: #333333;">++</span>)
pthread_create(<span style="color: #333333;">&</span>cons_thread[i], <span style="color: #007020;">NULL</span>, consumer, (<span style="color: #333399; font-weight: bold;">void</span><span style="color: #333333;">*</span>)<span style="color: #333333;">&</span>(b[i]));
<span style="color: #008800; font-weight: bold;">for</span> (<span style="color: #333399; font-weight: bold;">int</span> i <span style="color: #333333;">=</span> <span style="color: #0000dd; font-weight: bold;">0</span>; i <span style="color: #333333;"><</span> COUNT; i<span style="color: #333333;">++</span>)
{
pthread_join(prod_thread[i], <span style="color: #007020;">NULL</span>);
pthread_join(cons_thread[i], <span style="color: #007020;">NULL</span>);
}
}
</pre>
</td></tr>
</tbody></table>
</div>
<div>
<br />
<br />
Now you can compile the above snippet in with below command line option<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1</pre>
</td><td><pre style="line-height: 125%; margin: 0;">g++ -o a.out buffer.cpp -Wall -l pthread
</pre>
</td></tr>
</tbody></table>
</div>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Strangely, when you run the top report for the above program, it would look like below</div>
<div style="text-align: left;">
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8</pre>
</td><td><pre style="line-height: 125%; margin: 0;">top - <span style="color: blue;">13</span>:<span style="color: blue;">30</span>:<span style="color: blue;">31</span> up <span style="color: blue;">164</span> days, <span style="color: blue;">3</span>:<span style="color: blue;">59</span>, <span style="color: blue;">2</span> users, load average: <span style="color: blue;">0.83</span>, <span style="color: blue;">0.75</span>, <span style="color: blue;">0.39</span>
Tasks: <span style="color: blue;">1</span> total, <span style="color: blue;">0</span> running, <span style="color: blue;">1</span> sleeping, <span style="color: blue;">0</span> stopped, <span style="color: blue;">0</span> zombie
%Cpu(s): <span style="color: blue;">0.9</span> us, <span style="color: blue;">5.7</span> sy, <span style="color: blue;">0.0</span> ni, <span style="color: blue;">93.4</span> id, <span style="color: blue;">0.0</span> wa, <span style="color: blue;">0.0</span> hi, <span style="color: blue;">0.0</span> si, <span style="color: blue;">0.0</span> st
KiB Mem: <span style="color: blue;">32767840</span> total, <span style="color: blue;">28525000</span> used, <span style="color: blue;">4242840</span> free, <span style="color: blue;">519348</span> buffers
KiB Swap: <span style="color: blue;">32772092</span> total, <span style="color: blue;">187724</span> used, <span style="color: blue;">32584368</span> free <span style="color: blue;">25386684</span> cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
<span style="color: blue;">23453</span> root <span style="color: blue;">20</span> <span style="color: blue;">0</span> <span style="color: blue;">5786032</span> <span style="color: blue;">4420</span> <span style="color: blue;">2236</span> S <span style="color: blue;">237.8</span> <span style="color: blue;">0.0</span> <span style="color: blue;">0</span>:<span style="color: blue;">44.46</span> a.out
</pre>
</td></tr>
</tbody></table>
</div>
</div>
<div style="text-align: left;">
<br />
A strange thing you would identify is that the CPU taken by processor is 237.8 %.... Howdy!!!<br />
<i>Now let's see if our perf tool can find out the culprit....</i></div>
<h2 style="text-align: left;">
Running the tool</h2>
<div>
Command to run (I will explain more about the parameters in later blog) to record the profiler information</div>
<div>
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1</pre>
</td><td><pre style="line-height: 125%; margin: 0;">perf record --<span style="color: #008800; font-weight: bold;">call</span>-graph dwarf -F <span style="color: #6600ee; font-weight: bold;">99</span> ./a.out -g sleep <span style="color: #6600ee; font-weight: bold;">120</span>
</pre>
</td></tr>
</tbody></table>
</div>
<div>
<br /></div>
<div style="text-align: left;">
Command to peep-into the inner happenings in the tool</div>
<div style="text-align: left;">
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1</pre>
</td><td><pre style="line-height: 125%; margin: 0;">perf report -n -i perf.data
</pre>
</td></tr>
</tbody></table>
</div>
<div style="text-align: left;">
<br />
The output produced is.... - there goes my first clue!!!!<br />
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<table><tbody>
<tr><td><pre style="line-height: 125%; margin: 0;">1
2
3
4
5
6
7
8
9</pre>
</td><td><pre style="line-height: 125%; margin: 0;"> ┌─Warning:───────────────────────────────┐
│Processed <span style="color: #6600ee; font-weight: bold;">7149</span> events and lost <span style="color: #6600ee; font-weight: bold;">6</span> chunks!│
│ │
│Check IO/CPU overload! │
│ │
│ │
│ │
│Press any key... │
└────────────────────────────────────────┘
</pre>
</td></tr>
</tbody></table>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
It clearly tell us that there is a incorrect code-base which is causing a huge CPU overload (bingo it matches our top report).</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
Now, if you dis-assemble the report, you will get the below details</div>
<div style="text-align: left;">
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0px; text-align: left;">Samples: <span style="color: #0000dd; font-weight: bold;">6</span>K <span style="color: #008800; font-weight: bold;">of</span> <span style="color: #008800; font-weight: bold;">event</span> <span style="background-color: #fff0f0;">'cycles:pp'</span>, Event count (approx.): <span style="color: #0000dd; font-weight: bold;">76746912353054071</span>
Children Self Samples Command Shared Object Symbol <span style="background-color: #ffaaaa; color: red;">◆</span>
<span style="color: #333333;">+</span> <span style="color: #6600ee; font-weight: bold;">96.09</span><span style="color: #333333;">%</span> <span style="color: #6600ee; font-weight: bold;">0.09</span><span style="color: #333333;">%</span> <span style="color: #0000dd; font-weight: bold;">6</span> a.out a.out [.] consumer <span style="background-color: #ffaaaa; color: red;">▒</span>
<span style="color: #333333;">+</span> <span style="color: #6600ee; font-weight: bold;">92.53</span><span style="color: #333333;">%</span> <span style="color: #6600ee; font-weight: bold;">3.55</span><span style="color: #333333;">%</span> <span style="color: #0000dd; font-weight: bold;">30</span> a.out libc<span style="color: #333333;">-</span><span style="color: #6600ee; font-weight: bold;">2.19</span>.so [.] __GI___libc_nanosleep <span style="background-color: #ffaaaa; color: red;">▒</span>
<span style="color: #333333;">-</span> <span style="color: #6600ee; font-weight: bold;">92.37</span><span style="color: #333333;">%</span> <span style="color: #6600ee; font-weight: bold;">0.00</span><span style="color: #333333;">%</span> <span style="color: #0000dd; font-weight: bold;">1</span> a.out libc<span style="color: #333333;">-</span><span style="color: #6600ee; font-weight: bold;">2.19</span>.so [.] usleep <span style="background-color: #ffaaaa; color: red;">▒</span>
<span style="color: #333333;">-</span> usleep <span style="background-color: #ffaaaa; color: red;">▒</span>
<span style="color: #333333;">-</span> <span style="color: #6600ee; font-weight: bold;">92.37</span><span style="color: #333333;">%</span> consumer <span style="background-color: #ffaaaa; color: red;">▒</span>
<span style="color: #333333;">-</span> start_thread <span style="background-color: #ffaaaa; color: red;">▒</span>
__clone <span style="background-color: #ffaaaa; color: red;">▒</span>
<span style="color: #6600ee; font-weight: bold;">0.00</span><span style="color: #333333;">%</span> producer
</pre>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
hmmm... so consumer has a usleep which consumes almost all CPU.</div>
<div style="text-align: left;">
<br /></div>
<h3 style="text-align: left;">
Conclusion</h3>
<div style="text-align: left;">
</div>
<ul style="text-align: left;">
<li>The micro second sleep in consumer (callback function) is very ineffective.</li>
<li>Most of the CPU utilization is gone in sleep, wake-up from the kernel and again sleeping. This un-necessary ineffective code is causing us precious CPU cycles being wasted which otherwise could have been used for other purpose.</li>
</ul>
<div>
Extend this line of thinking a bit-further, assume you have such in-effective application running on AWS and you paying precious $$$ for the bad code. A HUGE loss of revenue which could be caught just by running the same application with this tool and save your life....<br />
<br />
Home-work : Run the same application, by changing the usleep(1) in consumer - Line 150 to usleep(1000000). You would see the change.</div>
<br />
<h2 style="text-align: left;">
hmmm, It looks interesting. What more can I do with it?</h2>
<div>
A lot more... wait for my next blog</div>
<div>
In the meanwhile, you can comment on any such similar free tools which can potentially catch such problem. :)<br />
<br />
<h2>
Further Reading...</h2>
</div>
<div>
<a href="https://techunravel.blogspot.com/2018/03/how-to-use-flame-graph.html" target="_blank">How to use flame-graph?</a><br />
<br />
<h2>
And a closing thought....</h2>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif9x7D5AMATzdimRfZrYLWA6BGbz0HHxesScQoX29C8UBBO4ubjvD1te3DUjow8waewOAdVCw_26AfG3xsdwB3e7z1YIvFTQlpmBxfgxK80VPtAZVK-XjGYPPlxMIJoNqyHq2zujYpEbGr/s1600/You+Win+You+Learn+Motivational+Poster.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="1132" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEif9x7D5AMATzdimRfZrYLWA6BGbz0HHxesScQoX29C8UBBO4ubjvD1te3DUjow8waewOAdVCw_26AfG3xsdwB3e7z1YIvFTQlpmBxfgxK80VPtAZVK-XjGYPPlxMIJoNqyHq2zujYpEbGr/s400/You+Win+You+Learn+Motivational+Poster.png" width="282" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-40596174480555511792011-10-29T20:59:00.000+05:302018-03-31T19:16:13.089+05:30Static/global variable and shared library behaviour<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">[Note : This would be a very long discussion and reader would need considerable amount of time to complete the reading]</span><br style="font-family: Arial,Helvetica,sans-serif;" /><b><br style="font-family: Arial,Helvetica,sans-serif;" /></b><span style="color: blue;"><b><span style="font-family: "arial" , "helvetica" , sans-serif;">Problem Description</span></b></span><u style="color: blue;"><br style="font-family: Arial,Helvetica,sans-serif;" /></u><span style="font-family: "arial" , "helvetica" , sans-serif;">Let's take an example of a class having a static variable or a global variable. </span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Header file :</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> x.h</span></span><br />
<pre><span style="font-size: small;">extern int g_s;</span>
<span style="font-size: small;">
<span style="font-family: "arial" , "helvetica" , sans-serif;">class XYZ</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">{</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"> static int s;</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"> private:</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"> static int getInt();</span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">};</span></span></pre>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Implementation file:</span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">x.cpp</span><br />
<br />
<pre><span style="font-size: small;">
int XYZ::s = 777;
int g_s = 7;
int XYZ::getInt()
{
return s;
}
</span>
</pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Now let us go through below questions (Little Quiz) and answer them</span><br />
<ul style="text-align: left;">
<li><span style="color: blue; font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">What is the scope of global variable (g_s) and static class variable (XYZ::s)?</span></span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Naturally, the answer would be global.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">For class static data member variable, the scope would be global with additional limitation of privilage (i.e all objects belonging to XYZ in the executable will have only one copy of s and only accessible for class interface).</span></li>
</ul>
</ul>
<ul style="text-align: left;">
<li><span style="color: blue; font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">How many copies of the variable exists?</span></span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">One</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
</ul>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Are the above answers really correct?</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> What happens in case of executable involving multiple shared libraries (so's) - each having object files of x (i.e x.o)?</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">We will explore these and come back questions (Little Quiz) and re-answer them in the end.</span><br />
<div>
<span style="font-size: small;"><br style="font-family: Arial,Helvetica,sans-serif;" /><b style="color: blue;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Description</span></b><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: "arial" , "helvetica" , sans-serif;">Consider that you are building and executable a.exe. </span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: "arial" , "helvetica" , sans-serif;">Now this executable consists of many objects and libraries - as shown in below diagram.</span></span><br />
<span style="font-size: small;"></span><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8-OOIDtCDbQ1c0NsfmSbY3D5Oa4QcKTlc2vI887LiIoQzBxe9mqwYFp6NC3zuiqi73qLogaEmkmA7JjFfnOvF5aAG6ZQEkWjwq58k02ID83BQt-mDe1M-xGZ0cbyQSACq0iwUSFQmXfRS/s1600/ExeWithLibs.bmp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8-OOIDtCDbQ1c0NsfmSbY3D5Oa4QcKTlc2vI887LiIoQzBxe9mqwYFp6NC3zuiqi73qLogaEmkmA7JjFfnOvF5aAG6ZQEkWjwq58k02ID83BQt-mDe1M-xGZ0cbyQSACq0iwUSFQmXfRS/s320/ExeWithLibs.bmp" width="246" /></a><span style="font-size: small;"><br style="font-family: Arial,Helvetica,sans-serif;" /></span><br />
<ol style="font-family: Arial,Helvetica,sans-serif; text-align: left;">
<li><span style="font-size: small;">libXYZ.so consists of 3 objects X.o, Y.o and Z.o </span></li>
<li><span style="font-size: small;">libVWX.so consists of 3 objects V.o, W.o and X.o</span><span style="font-size: small;"> </span></li>
<li><span style="font-size: small;">libWXY.so consists of 3 objects W.o, X.o and Y.o</span><span style="font-size: small;"> </span></li>
<li><span style="font-size: small;">and, the executable a.exe consists of </span><span style="font-size: small;"> </span></li>
<ol>
<li><span style="font-size: small;">objects X.o, M.o and A.o and </span></li>
<li><span style="font-size: small;">shared libraries (linked using -L/-l option in make file) libXYZ.so, libVWX.so and libWXY.so.</span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></li>
</ol>
</ol>
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">From the diagram,you can see that there are 4 copies of the x.o present in the final executable (represented by 4 copies of x.o object). Thus in the final executable, there are 4 copies of the static variable X::s in the system.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span></div>
<div>
<ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;">
<li style="color: blue;"><span style="font-size: small;">Does Building the executable like the above throw linker error during compilation (complaining of multiple definitions)?</span><span style="font-size: small;"> </span></li>
<ul>
<li><span style="font-size: small;">Surprisingly No!!!. From a detailed study I have done, the multiple definition is determined based on certain rule set - described below.</span></li>
</ul>
</ul>
<ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;"><ul>
<li><span style="font-size: small;">If the same symbol is found among the peers of objects and/or static library then it is flagged as error during compilation.For example</span></li>
</ul>
</ul>
<ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;"><ul><ul>
<li><span style="font-size: small;">if and only if X.o and M.o contains same symbol, then it is flagged as an error - in case of bulding executable.</span></li>
</ul>
</ul>
</ul>
<ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;"><ul><ul>
<li><span style="font-size: small;">if and only if X.o and Y.o contains same symbol, then it is flagged as an error - in case of bulding a shared library.</span><span style="font-size: small;"> </span></li>
</ul>
</ul>
</ul>
<ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;"><ul>
<li><span style="font-size: small;"> During linking (last stage of execution building), the shared library symbols are not resolved. That means, linker will not go into the symbol definitions present in any of the linker - in case it is already present in the object file of the executable.</span></li>
</ul>
<li><span style="color: blue; font-size: small;">Now let us come to the most interesting fact, In such circumstances, which symbol is used?</span><span style="font-size: small;"> </span></li>
<ul>
<li><span style="font-size: small;">It depends on the OS. </span></li>
</ul>
</ul>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;"> <b>SUSE 10 Linux</b>:</span><span style="font-size: small;"> </span></div>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;"> Here, the first encountered symbol is always placed in the Global Symbol Table (GST). <br />
So during execution symbol resolution from any of the library, the GST is dipped into and would always gets the first symbol - leading to always pointing to same copy of the variable. </span></div>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;"><br />
</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY4L8Js0gGvYzvEYp-eG7j_Pz9HVNmwaHdv9AlVgzikirKD5gAmee8MGQhV55ro1JLt_6nXZ-3XtkWojpU1jrJQhtegJt98te6XSoUWvIpfwBaPY8_ihn28IJ8mvM1W40ucwEi7sHIR9y6/s1600/SUSE+GST+Example.bmp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY4L8Js0gGvYzvEYp-eG7j_Pz9HVNmwaHdv9AlVgzikirKD5gAmee8MGQhV55ro1JLt_6nXZ-3XtkWojpU1jrJQhtegJt98te6XSoUWvIpfwBaPY8_ihn28IJ8mvM1W40ucwEi7sHIR9y6/s320/SUSE+GST+Example.bmp" width="320" /></a></div>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;"> <b>HP-UX:</b></span></div>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;"> Here, when the execution flows through a library, it would refer to its "global" variable in the scope of its library. This can be visualized as GST containing global variable with the resolution of the library name also.</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0nXg9AuY33nTRTaTw5QTcg1AsiY4gN9Uz9s0a1UGtTHJZJ0UC90sZekUbWsPfwXgrlun8FYANNX71YA0vJVaiS4weje3xC_eHR0RiyReRO4ztlOlMLGP1N2g37dn8tHDli_iDhRADQLsW/s1600/HPUX+GST+Example.bmp" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0nXg9AuY33nTRTaTw5QTcg1AsiY4gN9Uz9s0a1UGtTHJZJ0UC90sZekUbWsPfwXgrlun8FYANNX71YA0vJVaiS4weje3xC_eHR0RiyReRO4ztlOlMLGP1N2g37dn8tHDli_iDhRADQLsW/s320/HPUX+GST+Example.bmp" width="320" /></a></div>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;"> </span></div>
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> This is contrary to SUSE Linux.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">In short, SUSE Linux seems to have a single copy of reference for the global variable across all the libraries, overlapping objects in the executable. However, HPUX seems to have global copy for each library and it would be referenced during the execution.</span></span><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<ul style="text-align: left;">
<li><span style="color: blue; font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Doesn't it lead to different kind of behavior in both SUSE and HPUX OS for the same machine?</span></span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Yes, Try it out!!!</span> </li>
</ul>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"></span><br />
<ul style="text-align: left;">
<li><span style="color: blue; font-family: "arial" , "helvetica" , sans-serif;">How to avoid this kind of problem?</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"> The main problem in this example, is the use of global objects/static global variables in multiple shared libraries and also in the main object file. The general expectation is that the linker to complain the problem - but it does not. </span></li>
</ul>
</ul>
<span style="font-size: small;"><span style="color: blue; font-family: "arial" , "helvetica" , sans-serif;">So only other option is to change the libraries into static library and link to the executable, if no error is thrown, relink with shared libraries.</span><br style="color: blue; font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span><br style="font-family: Arial,Helvetica,sans-serif;" /><b style="color: blue;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Conclusion</span></b><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></span><br />
<span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Now let us come back to the little quiz (discussed in the beggining).</span></span><br />
<ul style="text-align: left;">
<li style="color: blue;"><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">What is the scope of global variable (g_s) and static class variable (XYZ::s)?</span></span></li>
<ul>
<li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">(Previous answer)</span></span></li>
<ul>
<li><span style="font-size: small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Naturally, the answer would be global.</span></span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">For class static data member variable, the scope would be global with additional limitation of privilage (i.e all objects belonging to XYZ in the executable will have only one copy of s).</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
</ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">(Now)</span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Global, with the scope of shared library (while building ".so") or executable.</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Symbol resolution is done at peer object (.o) or static library (.a). </span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">Recursive symbol resolution is not done among linked shared library to find out conflict resolution.</span></li>
</ul>
</ul>
<li><span style="color: blue; font-family: "arial" , "helvetica" , sans-serif;">How many copies of the variable exists?</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">(Previous answer) One</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;">(Now) One per the linkage scope (i.e shared library in this case, or executable for main object file)</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> </span></li>
</ul>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Regards,</span></div>
<div style="font-family: Arial,Helvetica,sans-serif;">
<span style="font-size: small;">Tech Unravel<br />
Supreme Debugging</span></div>
</div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com3tag:blogger.com,1999:blog-3662894856370166929.post-43325358846038087822011-10-04T23:48:00.000+05:302018-03-30T17:53:49.863+05:30[C++] Additional parenthesis are always safe! Really?<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="color: blue;">
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Background</span></span></h2>
</div>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">In general, a programmer would be advised to provide additional parenthesis in the code "when in doubts" with a presumption that it is safe. We will see on example here on how in certain scenarios it cannot be safe and programmer needs to be extra careful.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><span style="color: blue;"></span></span></span><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Consider this code (which does nothing but puts a map of class pointer index versus class pointer index)</span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><i>Sample 1</i> </span></span><br />
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">1: #include <iostream>
2: #include <unistd.h>
3: #include <map>
4: using namespace std;</code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.</code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.
51: int main()
52: {
53: const unsigned int MAX_COUNT = 20000;
54: map <I*,J*> contextHolder;
55: char *p[MAX_COUNT];
.</code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.</code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">. </code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">62: map <I*,J*>::iterator itr = contextHolder.begin();
63: map <I*,J*>::iterator itrEnd = contextHolder.end();
64: while(itr != itrEnd)
65: {
66: if(<b>someCondition</b>)
67: {
68: itr->first->f();
69: itr->second->f();
70: I* pTempI = itr->first;
71: J* pTempJ = itr->second;
72: delete pTempI;
73: delete pTempJ;
<span style="background-color: #cccccc; color: #660000;">74: <b>contextHolder.erase(itr); </b></span>
75: }
76: itr++;
77: } </code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.</code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.</code></pre>
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">.
82: return 0;
83: } </code></pre>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Now, let us discuss the code which is highlighted.</span></span><br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">contextHolder.erase(itr);</span></span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">This would lead to well known problem - crash,</span></span><br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Line 74 deletes the contents pointing to itr - thereby invalidating the pointer.</span></span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Line 76 increments the invalidated pointer - thus causing the crash.</span></span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">to solve this problem, developers are known to add the iteration increment during the erase operation itself. Like</span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><i>S</i>ample 2</span></span><br />
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">64: while(itr != itrEnd)
65: {
66: if(someCondition)
67: {
68: itr->first->f();
69: itr->second->f();
70: I* pTempI = itr->first;
71: J* pTempJ = itr->second;
72: delete pTempI;
73: delete pTempJ;
74: contextHolder.erase(itr++);
75: }
76: else
77: {
78: //do something else.
79: itr++;
80: }
81: } </code></pre>
<code></code><br />
<code></code><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">The fix in line 74 (In sample 2 versus sample 1) solves the problem with an increment operator for the iterator (because increment is done before invalidating the iterator). </span></span><br />
<br />
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><span style="color: blue;">Detailed Description</span></span></span></h2>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">All is well till now. But assume that a amateur programmer, who is confused about the working of increment operation in a complex statement like that of Sample 2, line 74 adds an addition braces for the increment iterator.</span></span><br />
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"> </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Sample 3</span></span><code style="color: black; word-wrap: normal;"></code><br />
<pre style="background: none repeat scroll 0% 0% rgb(240, 240, 240); border: 1px dashed rgb(204, 204, 204); color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">64: while(itr != itrEnd)
65: {
66: if(someCondition)
67: {
68: itr->first->f();
69: itr->second->f();
70: I* pTempI = itr->first;
71: J* pTempJ = itr->second;
72: delete pTempI;
73: delete pTempJ;
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;">74: contextHolder.erase<span style="font-size: x-large;">(<span style="color: red;">(</span></span>itr++<span style="font-size: x-large;"><span style="color: red;">)</span>)</span>; </span></span>
75: }
76: else
77: {
78: //do something else.
79: itr++;
80: }
81: </code></pre>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">We see that the application crashes at line 74.</span></span><br />
<br />
<div style="color: blue;">
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Analysis </span></span></h2>
</div>
<div style="color: blue;">
<br /></div>
<div style="color: blue;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><span style="color: black;">Now this seemingly innocent braces (highlighted in red) looks to do more harm than good. This is a classical coding problem - the iterator is extended (incremented) beyond the boundary (i.e itr.end() ) and then the map tries to erase the same element - "causing crash".</span></span></span></div>
<div style="color: blue;">
<br /></div>
<div style="color: blue;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><span style="color: black;">So adding this additional parenthesis lead to a disastrous bug. Moreover, hard to debug/difficult to reoccur.Any static check tool also will mostly will fail to unearth the problem.</span></span></span></div>
<div style="color: blue;">
<br /></div>
<div style="color: blue;">
<h2>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Take Away's </span></span></h2>
</div>
<div style="color: blue;">
<br /></div>
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"><span style="color: black;"></span>These kind of problems are very very difficult to catch at any cycle of SDLC phase. Even a reviewer (with immense experience) also will find difficult to catch these kind of issues.</span></span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"></span></span><br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">It is best to share these kind of unique issues which occur in projects as technical sharing across the organisation - so that developers are sensitised (adj : having an allergy or peculiar or excessive susceptibility :)), to such potential problem in code.</span></span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"></span></span><br />
<ul style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"> It is best for the developer not to blindly follow "any rule", but to understand the concept behind the logic and how compiler interprets the code - in this case - the priority of execution in the complex execution statement.</span></span></li>
</ul>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Regards,</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Tech Unravel</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;">Supreme Debugging</span></span><br />
<div>
<code style="color: black; word-wrap: normal;"> </code><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: small;"></span></span></div>
</div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-23043549634178286362011-09-01T20:23:00.000+05:302011-09-01T20:27:52.971+05:30C++ Static data member variable and its initialization<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-size: small;"><b style="color: blue;"><span style="font-family: Arial,Helvetica,sans-serif;">Question</span></b><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">With regard to static data member initialization, as we are all aware of the syntax used in c++, a sample example is provided below </span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;"></span></span><br />
<pre><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">class Account
<span style="font-family: Arial,Helvetica,sans-serif;">{</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> public:</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> static double rate() { return interestRate; }</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> static void rate(double); // sets a new rate</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> private:</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> static const int period = 30; // interest posted every 30 days</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> static double interestRate;</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> double daily_tbl[period]; // ok: period is constant expression</span>
<span style="font-family: Arial,Helvetica,sans-serif;">};</span>
<span style="font-family: Arial,Helvetica,sans-serif;">// define and initialize static class member</span>
<span style="font-family: Arial,Helvetica,sans-serif;">double Account::interestRate = initRate();</span>
<span style="font-family: Arial,Helvetica,sans-serif;">int main()</span>
<span style="font-family: Arial,Helvetica,sans-serif;">{</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> Account ac;</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> ...</span>
<span style="font-family: Arial,Helvetica,sans-serif;"> ...</span>
<span style="font-family: Arial,Helvetica,sans-serif;">}</span></span></span></pre>
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">Now here we see two kinds of initialization of static member variable.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">a. The const data member variable is initialized at the time of declaration.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">b. The non-const data member variable is initialized outside the class declaration. </span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">My initial view was - this is strange!!!</span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">Is it still retained in language because of legacy mistake? Or is there really a need to have such kind of differentiation?</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><b style="color: blue; font-family: Arial,Helvetica,sans-serif;">Answer</b><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">I did ponder over this question and posted the same in a popular Q&A portal </span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">(stackoverflow).</span><span style="font-family: Arial,Helvetica,sans-serif;">You can find the original question with detailed reply by members <a href="http://stackoverflow.com/questions/4547660/c-static-member-variable-and-its-initialization">here.</a></span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;"></span><span style="font-family: Arial,Helvetica,sans-serif;"> </span></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Let me start re-discussion the same question in a more detailed fashion.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">One of the first pointer is ODR (<a href="http://en.wikipedia.org/wiki/One_Definition_Rule">Wikipedia One Definition Rule</a>).</span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">a. This states that any translation unit - a template/type/function or an object can have no more than one definition.</span><span style="font-family: Arial,Helvetica,sans-serif;">since a definition provides the instance, it can be only one in a translation unit. However, declarations can be many.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">b. The same argument extends for the exceutable. In an executable, there can be only one definition for a variable (after scope </span><span style="font-family: Arial,Helvetica,sans-serif;">resolution) and function definitions.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">This answers the second question. </span></span><br />
</span><br />
<ul style="text-align: left;"><span style="font-size: small;">
<li><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">If non-const static member was allowed to be defined in the class itself,then it would have been</span></span><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"> impossible to include these class declaration in different translation units. </span></span></li>
</span></ul>
<span style="font-size: small;">
<ul style="text-align: left;">
<li><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Thus to avoid violation of ODR rule,</span><span style="font-family: Arial,Helvetica,sans-serif;">the non-const static data member must be defined outside the class declaration.</span></span></li>
</ul>
<span style="font-size: small;"><br style="font-family: Arial,Helvetica,sans-serif;" /></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Now let us come to the first part of the question (which can be explained much more easily), a const data member can be defined inside </span><span style="font-family: Arial,Helvetica,sans-serif;">the class member for two reasons - </span></span><br />
<ul style="text-align: left;">
<li><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"></span><span style="font-family: Arial,Helvetica,sans-serif;">Even if multiple copies are present in each translation unit, the expression will be evaluated during compile time and does not</span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"> </span>violate the ODR theme.</span></li>
</ul>
<ul style="text-align: left;">
<li><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">A const must be defined during the creation of the instance (Since it cannot be changed later), if not for the above rule, there</span></span><span style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"> </span>would have been chance to change it on every object instansiation or having different definition in each translation unit.</span></li>
</ul>
<span style="font-size: small;"><br style="font-family: Arial,Helvetica,sans-serif;" /><b style="color: blue;"><span style="font-family: Arial,Helvetica,sans-serif;">Take Away's</span></b></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"> </span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">Developers need not remember each language syntax and its subtle changes based on memorizing the scenarios under which error are thrown by compilers.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;"> </span></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Instead simple interpretation and logical understanding of the constraints that forced to structure the language can be studied and thus understand the syntax by logical interpretation.</span></span><br />
<br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Regards,</span></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Tech Unravel,</span></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">Supreme Debugging.</span></span></span></div>
Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-15355231603768210952011-08-26T23:07:00.000+05:302011-09-01T12:24:50.338+05:30Win32 Env variable Pitfall of mismatching SetEnvironmentVariable and getEnv<div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;"><b style="color: blue;">Problem Description</b> </span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">Below article will give a brief introduction on environment variables and one strange (for a Unix programmer) behavior in WIN32. </span></span><br />
<span style="font-size: small;"><span style="font-family: Arial,Helvetica,sans-serif;">i.e getEnv() does not retrieve some environment variable but retrieve some in WIN32. But in Unix, all environment variables are retrieved.</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><b style="color: blue;"><span style="font-family: Arial,Helvetica,sans-serif;">Background </span></b><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">As quoted in wiki - </span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">Environment variables are a set of dynamically named values that can affect the way running processes will behave. (</span><a href="http://en.wikipedia.org/wiki/Environment_variable" style="font-family: Arial,Helvetica,sans-serif;">http://en.wikipedia.org/wiki/Environment_variable</a><span style="font-family: Arial,Helvetica,sans-serif;"> )</span><br style="font-family: Arial,Helvetica,sans-serif;" /><br style="font-family: Arial,Helvetica,sans-serif;" /><span style="font-family: Arial,Helvetica,sans-serif;">In Unix environment, it is usual to set environment variables in 2 ways - </span></span><br />
<ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;"><li><span style="font-size: small;">Type 1 setting : Use setenv command (generally collect these command into a environment sourcing file) and call it to set all the environment variables.</span></li>
</ul><ul style="font-family: Arial,Helvetica,sans-serif; text-align: left;"><li><span style="font-size: small;">Type 2 setting : Use setenv API provided in the stdlib library and set the environment variables. API Info - <a href="http://pubs.opengroup.org/onlinepubs/007904975/functions/setenv.html">http://pubs.opengroup.org/onlinepubs/007904975/functions/setenv.html</a></span></li>
</ul><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"> </span><span style="font-size: small;">A point of note : The first option is executed from the shell command and all these environment variables are inherited by the process - it is more of a statically set environment variable. The second option is a more dynamic option which can be set during the process execution.<br />
<br />
These variables, thus set can be accessed with getEnv ( </span><span style="font-size: small;"><a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/getenv.html">http://pubs.opengroup.org/onlinepubs/009695399/functions/getenv.html</a>) again available from stdlib.<br />
<br />
Now, let us come to the crux of the problem. <br />
<br />
In Unix, both the environment variables set by "Type 1" and "Type 2" can be accessed by getEnv function. When this code is ported in WIN32, it is strangely seen that statically set environment variables only will be accessible for getEnv function. <br />
<br />
Any environment variable set using "Type2" setting (i.e dynamically set variable) using setEnvironmentVariable API (<a href="http://msdn.microsoft.com/en-us/library/ms686206%28v=vs.85%29.aspx">http://msdn.microsoft.com/en-us/library/ms686206%28v=vs.85%29.aspx</a>) cannot be accessed by getEnv() function.<br />
<br />
<b style="color: blue;">Example</b> </span></div><div style="font-family: Arial,Helvetica,sans-serif;"></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Let us take a look at below code</span></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Example 1 </span></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">(In command line, setenv LIB="c:\\mylib;c:\\yourlib" is already done)-</span></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br />
</span><br />
<blockquote><pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">1: #include <stdlib.h>
2: #include <stdio.h>
3: int main( void )
4: {
5: char *libvar;
6: /* Get new value. */
7: libvar = getenv( "LIB" );
8: if( libvar != NULL )
9: printf( "New LIB variable is: %s\n", libvar );
10: return 0;
11: } </code></pre></blockquote><span style="font-size: small;">In the above case, we will be able to get the output with detailed environment variable set from command line.</span><br />
<br />
<span style="font-size: small;">However, the below example has a start change in behavior </span></div><div style="font-family: Arial,Helvetica,sans-serif;"></div><div style="font-family: Arial,Helvetica,sans-serif;"><div><span style="font-size: small;">Example 2</span> </div><div><span style="font-size: small;">(In command line, LIB="c:\\mylib;c:\\yourlib" is not set from command line, but set in program)-</span></div><span style="font-size: small;"> </span><br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">1: #include <stdlib.h>
2: #include <stdio.h>
3: int main( void )
4: {
5: char *libvar;
6: SetEnvironmentVariable("LIB", "C:\WIN32");
7: /* Get new value. */
8: libvar = getenv( "LIB" );
9: if( libvar != NULL )
10: printf( "New LIB variable is: %s\n", libvar );
11: return 0;
12: } </code></pre><blockquote><span style="font-size: small;">Here, the output would not be printed, owing to the fact that Type 2 setting env variables are not obtained by the getenv() API.</span></blockquote></div><div style="font-family: Arial,Helvetica,sans-serif;"></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br />
</span></div><div style="color: blue; font-family: Arial,Helvetica,sans-serif;"><b><span style="font-size: small;">Root cause</span></b></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">The getEnv() function provides the output only from the internal cache (data structures) populated during the process start-up i.e the environment variable set in the shell. This is mentioned in the MSDN manual -</span></div><div style="font-family: Arial,Helvetica,sans-serif;"><blockquote><span style="font-size: small;"> </span><span style="font-size: small;"><b style="background-color: #cccccc;">getenv</b><span style="background-color: #cccccc;"> operates only on the data structures accessible to the run-time library and not on the environment "segment" created for the process by the operating system</span></span></blockquote><span style="font-size: small;">Thus in WIN32, getEnv() will not reflect the environment variable set during the execution of the program. Instead users need to use GetEnvironmentVariableA() API provided by MSDN.</span></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br />
</span></div><div style="color: blue; font-family: Arial,Helvetica,sans-serif;"><b><span style="font-size: small;">Take Away's</span></b></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;"> </span></div><div style="font-family: Arial,Helvetica,sans-serif;"><ul style="text-align: left;"><li><span style="font-size: small;">Developers needs to be very discreet in using API's and also checking for differing "meanings" across different OS.</span></li>
</ul></div><div style="font-family: Arial,Helvetica,sans-serif;"><ul style="text-align: left;"><li><span style="font-size: small;">The manual should be read in detail and "in the corner" mentioning of the difference in behavior must be checked with a magnifying lens.</span></li>
</ul><span style="font-size: small;">Regards, </span></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Tech Unravel.</span></div><div style="font-family: Arial,Helvetica,sans-serif;"><span style="font-size: small;">Supreme Debugging </span></div></div>Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0tag:blogger.com,1999:blog-3662894856370166929.post-77562099844417949072011-08-19T23:35:00.000+05:302011-09-01T12:27:18.552+05:30[Windows Programming] : Exception Generation during process creation or system command execution<div dir="ltr" style="text-align: left;" trbidi="on"><span style="font-size: 85%;"><span style="color: black; font-family: arial;"><span style="color: #3333ff; font-style: italic; font-weight: bold;"> <br />
Problem Description <br />
<br />
</span></span> <br />
<span style="color: black; font-family: arial;">During my windows porting activity (Existing code base in *ix - CPP programming language) <br />
I came across a very strange problem <br />
<br />
The project code bases uses many system command for various activity, <br />
and one such operation is to untar (unrar in windows) the package to be launched. <br />
<br />
Strangely the problem faced was that the unrar command used was not working as expected in XP SP3 patch. <br />
Instead for both error and success scenario strange error code was displayed. <br />
Below steps provides an insight with the investigation steps and the final end result. <br />
<br />
<br />
<span style="color: #3333ff; font-style: italic; font-weight: bold;">Investigation Steps</span> <br />
<br />
Step 1: The initial suspect was related to compatibility of system command in Windows. <br />
Unlike *ix, where the command is very well defined, it was hard to find out a suitable documentation for the same in <br />
windows. <br />
<br />
So immediately suspect was API issue (compatibility problem). <br />
<br />
</span><span style="font-family: arial;"><span style="color: black;">Step 2: Next the focus shifted to an well established, command execution methodology. Immediate reference came</span> <br />
<span style="color: black;">from MSDN content -</span> <br />
<br />
</span><a href="http://msdn.microsoft.com/en-us/library/ms682425%28v=vs.85%29.aspx"><span style="font-family: arial;">http://msdn.microsoft.com/en-us/library/ms682425%28v=vs.85%29.aspx</span></a> <br />
<br />
<span style="color: black; font-family: arial;">Step 3: A sample program was derived - which used the createProcess API was used for untarring. <br />
</span><span style="font-family: arial;"> <br />
<span style="color: black; font-style: italic;">Sample code looked as shown below</span> <br />
</span></span><br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">1: int main()
2: {
3: string ProgramName = "C:\\Program Files\\WinRAR\\WinRAR.exe";
4: STARTUPINFO StartupInfo;
5: PROCESS_INFORMATION ProcessInfo;
6: memset(&StartupInfo, 0, sizeof(STARTUPINFO));
7: memset(&ProcessInfo, 0, sizeof(PROCESS_INFORMATION);
8: if (CreateProcess((LPCTSTR)ProgramName.c_str(),(LPCTSTR)"WinRAR.exe x -y -ibck d:\\abc.tar d:\\"),NULL,
9: NULL,
10: FALSE,
11: NORMAL_PRIORITY_CLASS,
12: NULL,
13: NULL,
14: &StartupInfo,
15: &ProcessInfo) == 0)
16: {
17: string tmpStr("Error executing");
18: tmpStr += ProgramName;
19: cout<<"StmtDesigner"<<<"createprocess dword="" exitcode="0;" if="" span="" string="" tmpstr="">
20: }
21: CloseHandle(ProcessInfo.hProcess);
22: CloseHandle(ProcessInfo.hThread);
23: getch();
24: return 0;
25: }</code></pre><span style="font-size: 85%;"><span style="color: #663333; font-family: arial;"><tmpstr<<"createprocess dword="" exitcode="0;" if="" span="" string="" tmpstr=""><span style="font-family: arial;"><span style="color: #663333;"></span> <br />
<br />
<br />
</span><span style="color: black; font-family: arial;">In this case strangely, </span> <br />
<span style="color: black; font-family: arial;">SP2 provided expected result - i.e 0 for success and 1 (non-zero) for failure.</span> <br />
<span style="color: black; font-family: arial;">SP3 provided unexpected result - i.e 3221225477 for both success and failure. <br />
<br />
Step 4: <br />
The exit code value - 3221225477 was very critical, now when this was transformed <br />
into HEX,it represented - C0000005 - which represents access violation in windows. <br />
Now the investigation took a turn on what could have lead to the problem. <br />
<br />
Step 5: <br />
With lot of google'ing and detailed search, It was found that a good utility <br />
called "Process Monitor" would mimic the functionality of strace in Unix. <br />
<br />
</span><a href="http://technet.microsoft.com/en-us/sysinternals/bb896645"><span style="font-family: arial;">http://technet.microsoft.com/en-us/sysinternals/bb896645</span> <br />
</a> <br />
<span style="color: black; font-family: arial;">Was downloaded and on SP3 machine and the sample program above was run with <br />
monitor tool. <br />
<br />
Step 6: <br />
Process Monitor helped (with very good filtering feature) to find out that <br />
WinRar process exited with exception. <br />
<br />
This was noted as an event in the Process Monitor, <br />
Further analysis of the stack during the Winrar exit event showed that the <br />
$WINDOWS/system32/ntdll.dll interacted with one of the proprietary library present <br />
in the same path thus the exit status was not captured correctly as ntdll.dll <br />
was trying to access some function of proprietary library - causing access violation. <br />
<br />
This was cross confirmed by SP2 machine - which did not have this dll file <br />
(and program</span><span style="color: black;"> </span><span style="color: black; font-family: arial;">was executed successfully). <br />
<br />
Also after renaming/removing the proprietary library from system32 directory, <br />
the operation succeeded even in SP3 operation (both createProcess and system command) <br />
<br />
<span style="color: #3333ff; font-style: italic; font-weight: bold;">Take Away's</span> <br />
<br />
1. It is still unclear why the presence of proprietary application library in <br />
$WINDOWS/system32 caused ntdll.dll to fail thread/process exit for WinRar application. <br />
<br />
2. Exceptions are also thrown by the API's GetExitCodeProcess. <br />
So it is not restricted to a range of values. <br />
<br />
3. Placing proprietary application in the system32 location was not a good idea. <br />
No thought on its impact was done. <br />
<br />
4. Process Monitor is an excellent tool and is one of the must haves for the windows </span> <br />
<span style="color: black; font-family: arial;">developer. <br />
<br />
</span><span style="color: black; font-family: arial;">5. Whenever C0000005 and other exceptions are the outcome of a program, program and <br />
</span><span style="color: black; font-family: arial;">its system dependency must be thoroughly checked for all kinds of violation <br />
</span><span style="color: black; font-family: arial;">(In this case an unwanted library causing some kind of interference). <br />
<br />
</span><span style="color: black; font-family: arial;">6. Initial suspect on SP2/SP3 difference proved futile and invalid, </span> <br />
<span style="color: black; font-family: arial;">afterall looks like windows is maintaining good backward compatibility :). <br />
<br />
</span> <br />
<span style="color: black; font-family: arial;">Regards, <br />
</span><span style="color: black; font-family: arial;">Tech Unravel <br />
</span><span style="color: black; font-family: arial;">Supreme Debugging</span> <br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
</tmpstr<<"createprocess></span></span></div>Kiranhttp://www.blogger.com/profile/14679685438891877803noreply@blogger.com0