1 17:18:19 * stevenknight (n=stevenkn@nat/google/x-ff6fd1a26f0686e6) has joined #scons
2 17:25:16 * Jason_ (n=Jason@bementil-116.illinois.prairieinet.net) has joined #scons
3 17:29:26 * garyo-home (n=chatzill@209.6.158.38) has joined #scons
4 17:31:26 * GregoryNoel is here
5 17:31:38 <GregoryNoel> Hi, guys, are we ready?
6 17:31:51 <garyo-home> Hi Greg. Ready as I'll be. :-/
7 17:32:23 <GregoryNoel> I only see limited comments in the spreadsheet; it's gonna be a slow night.
8 17:32:53 <garyo-home> yes, sorry. Just doing some now.
9 17:33:02 <stevenknight> hey
10 17:33:07 <stevenknight> me too
11 17:34:15 <GregoryNoel> 1945?
12 17:34:42 <garyo-home> steven to research & pick one option.
13 17:34:47 <garyo-home> w/ Ludwig.
14 17:34:57 <garyo-home> IMHO.
15 17:35:25 <GregoryNoel> I'll go with that, at least for now. We need to come up with something better in the long run.
16 17:36:17 <garyo-home> Not sure about your comment re: rewriting implicit dependency logic, but won't go there now.
17 17:36:44 <stevenknight> agree w/what you guys are saying
18 17:37:17 <GregoryNoel> I think we can do more than what we are now by caching negatives, but it's not the time to redesign it.
19 17:37:36 <garyo-home> right.
20 17:38:00 <GregoryNoel> so research, Steven + Ludwig?
21 17:38:11 <garyo-home> good.
22 17:38:19 <GregoryNoel> And once the decision is reached, Ludwig to implement?
23 17:38:28 <stevenknight> done
24 17:38:31 <GregoryNoel> done
25 17:38:35 <GregoryNoel> 2258
26 17:38:36 <garyo-home> 2258: stevenknight to add a hook to get help text, 2.x p4?
27 17:39:07 <GregoryNoel> Hmmmm.....
28 17:40:06 <GregoryNoel> Yeah, something like that. His suggestion is awkward, but it should be possible to work out something.
29 17:40:38 <GregoryNoel> And for 2.x p4, we don't need to volunteer anyone.
30 17:40:41 <GregoryNoel> yet
31 17:40:48 <garyo-home> agree.
32 17:40:52 <stevenknight> okay, i'm back from the spreadsheet
33 17:41:04 <stevenknight> 2258: done
34 17:41:09 <stevenknight> 2.x p4, future draft pick
35 17:41:10 <GregoryNoel> 2261
36 17:41:38 <stevenknight> it's a google guy, so i'd like a crack at it
37 17:41:58 <GregoryNoel> We have another issue looking at this already; one should be closed.
38 17:42:00 <garyo-home> Shouldn't he be using the target filename as the target? (And not rm/mkdir either)
39 17:41:59 <stevenknight> if i can't come up with a real case, i'll mark it invalid
40 17:42:35 <garyo-home> ok, that's good enough for me. Steven to find testcase or mark invalid.
41 17:43:05 <GregoryNoel> or dup to the other one
42 17:43:16 <stevenknight> yeah, i guess i view it similar to unpacking a .tar.gz into a directory
43 17:43:22 <stevenknight> i think you should be able to specify the directory
44 17:43:30 <stevenknight> and have it do the right thing w.r.t. changes to the source(s)
45 17:43:41 <stevenknight> but i'll mark it invalid if I'm just dreaming
46 17:43:47 <GregoryNoel> done
47 17:44:00 <GregoryNoel> 2262
48 17:44:12 <stevenknight> i think it should be trivial
49 17:45:26 <GregoryNoel> If you think it's trivial, I'll let you have it.
50 17:45:20 <stevenknight> 2262: anytime, anyone
51 17:45:29 <GregoryNoel> anytime is fine
52 17:45:39 <garyo-home> The OP says he'd like "not .py" but I think .py makes the most sense.
53 17:45:41 <stevenknight> okay, 2262 anytime stevenknight
54 17:46:15 <Jason_> I like the .scons ,myself
55 17:46:17 <stevenknight> while we're at it, i'd go for both .py and .scons
56 17:46:36 <stevenknight> and draw the line there
57 17:46:38 <GregoryNoel> I've also seen .sc used.
58 17:46:45 <garyo-home> ok, why not. Just pick an order, seems fine to me.
59 17:47:04 <garyo-home> Greg: I think .sc is too short.
60 17:47:17 <GregoryNoel> Sigh. Seems too complex; I'd still wontfix it as I indicated.
61 17:47:02 <stevenknight> isn't .sc for SuperCalc files...?
62 17:47:05 * stevenknight shows his age...
63 17:47:12 <garyo-home> SuperCalc?!!
64 17:47:38 <GregoryNoel> Yeah, I remember it, too.
65 17:47:46 * GregoryNoel _really_ shows his age.
66 17:47:47 <stevenknight> 2262: anytime, stevenknight
67 17:47:52 <GregoryNoel> done
68 17:48:03 <GregoryNoel> 2264
69 17:48:27 <garyo-home> Greg, you looked at this most -- is the order reversal the only thing going on here?
70 17:48:28 <stevenknight> 2.x p2 gregnoel?
71 17:49:16 <GregoryNoel> No, the dependencies aren't reversed, reversing the order of the names on the command line produces different trees.
72 17:49:43 <garyo-home> ok, I believe you :-)
73 17:49:47 <GregoryNoel> No matter which way you list the names on the command line, it should show the same dependencies.
74 17:49:57 <garyo-home> do you mind taking it?
75 17:50:45 <GregoryNoel> I know almost nothing about that part of the code, so I'd rather not, but if you push, I'll consider it.
76 17:51:04 <stevenknight> 2.x p2 can be future draft pick
77 17:51:16 <stevenknight> or else put my name on it
78 17:51:16 <GregoryNoel> I'll go with that
79 17:51:27 <stevenknight> okay, 2.x p2 future draft pick
80 17:51:28 <stevenknight> done
81 17:51:40 <GregoryNoel> 2265
82 17:51:50 <garyo-home> ah, yes.
83 17:51:56 <Jason_> just so it is known this is not new
84 17:52:15 <stevenknight> 2265: 1.2 p1 stevenknight
85 17:52:16 <garyo-home> Jason_, you mean 2265?
86 17:52:23 <Jason_> Yes
87 17:52:27 <stevenknight> like I said, I'm just in Taskmaster lately
88 17:52:28 <Jason_> I filed it
89 17:52:30 <garyo-home> thanks, Steven.
90 17:52:47 <garyo-home> Ah, you're *that* Jason -- welcome!
91 17:52:55 <Jason_> Thanks
92 17:53:13 <GregoryNoel> stevenknight, done
93 17:53:19 <GregoryNoel> and thanks
94 17:53:18 <stevenknight> Jason_: glad to have you here
95 17:53:24 <Jason_> hope we can talk latter :-)
96 17:53:38 <stevenknight> yep
97 17:53:46 <stevenknight> i have a few non-issue topics i'd like to discuss as well
98 17:53:54 <garyo-home> ok
99 17:54:12 <GregoryNoel> 2266: I'll let you guys fight it out
100 17:54:17 <stevenknight> 2266: 1.x p3 stevenknight
101 17:54:36 <Jason_> we have seen this .. it depends on the python used
102 17:54:43 <garyo-home> 2266: not sure why this doesn't work for him. I do this all the time. What python is he using?
103 17:54:58 <garyo-home> Do NOT use cygwin python.
104 17:55:15 <Jason_> That follows what we see here as well
105 17:55:29 <Jason_> cygwin python will mess up a windows based build
106 17:55:37 <garyo-home> 2266: let me take it and suggest that to him.
107 17:55:43 <GregoryNoel> done
108 17:55:54 <GregoryNoel> 2267
109 17:56:47 <stevenknight> either 1.2 and defer if investigation shows we can live with it
110 17:56:56 <GregoryNoel> This is the new Action.py code I put in, but I can't see how it's not initialized.
111 17:56:57 <stevenknight> or resesarch with the option of making it a 1.2 blocker
112 17:58:19 <GregoryNoel> Seeing no hands, I'll research it. It's a bad week for that, but I'll try to make time.
113 17:58:22 <garyo-home> Greg: possible exception issue?
114 17:58:27 <GregoryNoel> garyo-home, huh?
115 17:58:35 <garyo-home> never mind.
116 17:59:15 <GregoryNoel> so 2267 research me?
117 17:59:20 <stevenknight> done
118 17:59:47 <stevenknight> GregoryNoel: let me know if I can help, I've had to debug things like this many times
119 17:59:53 <GregoryNoel> thanks
120 18:00:05 <GregoryNoel> Before we go on, can I insert a topic?
121 18:00:17 <garyo-home> sure
122 18:00:34 <stevenknight> sure, let's do GN's first, then Jason_, then me
123 18:00:30 <GregoryNoel> I'd like to change the agenda slightly. I'd like to add a permanent item to discuss the current status of the release schedule and how we're doing on it.
124 18:00:30 <GregoryNoel> Discussion points would be what's expected in the next release(s) (whether checkpoint, candidate, or public), adjustments to the schedule, whether we should do any retriage for current or future releases, things like that.
125 18:00:30 <GregoryNoel> I'd put it between the current issues and the backlog. (Aside: we're almost to the home stretch for finishing off the backlog (hopefully we'll kill off 2005 this time or next time); after that, the time now spent on backlog will be taken up by bickering about retriage of issues, so we should discuss the release schedule strategy before the tactics. This positioning puts it in the right place for the future.)
126 18:01:13 <stevenknight> +1 to discussing release schedule regularly
127 18:01:24 <garyo-home> greg: +1 to your agenda change. I always review the schedule before the meeting anyway.
128 18:02:16 <GregoryNoel> OK, I'll do it. Since the slot would be now, should we discuss the schedule now?
129 18:02:21 <stevenknight> yes
130 18:02:23 <garyo-home> yes.
131 18:02:31 <GregoryNoel> Schedule: 1.2 was due out 24 November. Needless to say, we didn't make it. And issues 2265 and 2267 could be a blockers; we may need to slip 1.2 until that's resolved.
132 18:02:31 <GregoryNoel> We were supposed to have a checkpoint out 1 December. We didn't make that, either.
133 18:02:31 <GregoryNoel> Assuming we get the release out this week, we have some decisions to make. Should we slide the rest of the schedule by a week, so the next checkpoint is 8 December? Cancel the 1 December checkpoint and keep the current schedule, with the next checkpoint 15 December? Or something else?
134 18:02:31 <GregoryNoel> I lean toward dropping the 1 December checkpoint, but I could be easily persuaded differently.
135 18:02:59 <stevenknight> i put out a release candidate 11/25?
136 18:03:33 <GregoryNoel> I think that's about right
137 18:03:28 <garyo-home> I have three 1.2 tickets on my plate. None serious except maybe #2048 which I'm working on but fails in weird ways
138 18:04:42 <stevenknight> oh, wait, I see--the release schedule is 12/1 checkpoint that was supposed to be *after* 1.2 was out
139 18:04:45 <stevenknight> and i'm late with 1.2
140 18:04:51 <stevenknight> right?
141 18:05:10 <garyo-home> I think so. I'd like to get 1.2 out soon then put vs_revamp in for next checkpoint after that.
142 18:05:40 <GregoryNoel> stevenknight, yes
143 18:05:37 <stevenknight> right, and I actually have batched Visual C/C++ compilation working on a private branch that I'd like to go in
144 18:05:56 <garyo-home> wow!
145 18:05:57 <Jason_> I would like to add stuff to the revamp
146 18:06:12 <garyo-home> Jason: good. You have an intelc.py version of it too I think.
147 18:06:12 <stevenknight> okay, so let's review 1.2 issues and push anything that's not a potential/likely blocker in 1.3
148 18:06:18 <Jason_> We have some work not finished that will improve it and make it faster
149 18:06:38 <garyo-home> OK. Let's get it into the trunk, then review & apply your changes
150 18:06:46 <Jason_> yep... still working on that part
151 18:06:48 <garyo-home> ok?
152 18:07:02 <Jason_> plus we have early support for vs 2010 and intel 11
153 18:07:02 <stevenknight> garyo_home: "it" into the trunk: "it" == vs_revamp?
154 18:07:14 <stevenknight> Jason_: sweet
155 18:07:31 <garyo-home> steven: yes, it==vs_revamp.
156 18:07:36 <Jason_> I have "people" that know "people"
157 18:07:38 <stevenknight> okay, agreed
158 18:08:01 <Jason_> Well my fixes to the revamp.. are a bit different from the work David did
159 18:08:02 <garyo-home> There are 40 open 1.2 issues.
160 18:08:05 <GregoryNoel> Almost 40 issues scheduled for 1.2 still
161 18:08:13 <GregoryNoel> garyo-home, jinx
162 18:08:47 <garyo-home> Jason_: send a summary of your ideas & differences to the list, we'll discuss it there.
163 18:08:55 <Jason_> ok
164 18:09:13 <garyo-home> It's a bigger forum, David will be present, etc.
165 18:09:16 <stevenknight> a lot of the 40 issues are readily rollable to 1.3
166 18:09:19 <stevenknight> doc issues, e.g.
167 18:09:57 <GregoryNoel> Mine are all doc and testing (I haven't figured out how to test Action._subproc yet)
168 18:10:05 <garyo-home> 15 P2s, the rest P3 & P4.
169 18:10:49 <stevenknight> oh, and 40 doesn't count the two potential blockers we discussed tonight
170 18:11:15 <stevenknight> quick, quick glance suggests all P3 & P4 are automatically deferrable
171 18:12:07 <garyo-home> agreed.
172 18:12:09 <stevenknight> and I think the cournape P2 issues are because we once thought vs_revamp was going to make it into 1.2
173 18:12:54 <garyo-home> Is Benoit around? Any likelihood of his 4 issues getting done?
174 18:13:06 <stevenknight> slim, he's been AWOL for a while
175 18:13:21 <garyo-home> thought so.
176 18:13:31 <stevenknight> worth considering reassigning his issues so they don't languish
177 18:14:04 <garyo-home> Yes. I suggest moving to 1.3 for now, then reassigning.
178 18:15:02 <GregoryNoel> garyo-home, agree
179 18:15:13 <stevenknight> okay, so it sounds like the only two issues that don't go to 1.3 are the two new potential blockers
180 18:15:32 <GregoryNoel> I can do that with a mass move.
181 18:15:41 <stevenknight> GregoryNoel: thanks
182 18:15:33 <stevenknight> we get patches for those (or research suggests deferring to 1.3) and then I cut another 1.2 candidate
183 18:16:02 <garyo-home> What's up w/ 2249 though?
184 18:16:44 <stevenknight> dunno...
185 18:16:47 <garyo-home> (Not that it should be in 1.2 anyway, but how'd it get there?)
186 18:17:07 <stevenknight> vs_revamp, we thought it would be in 1.2
187 18:17:08 <garyo-home> It can move to 1.3 too since it's vs_revamp related.
188 18:17:59 <garyo-home> ok, so given all that, what's the proposed 1.2 release date?
189 18:18:10 <GregoryNoel> ASAP?
190 18:18:13 <stevenknight> stake in the ground: 12 December, a week from Friday
191 18:18:20 <stevenknight> candidate 5 December, two days
192 18:18:32 <stevenknight> for the two potential blockers
193 18:18:41 <GregoryNoel> works; I'll update the roadmap
194 18:18:44 <garyo-home> ok, so I have some time to sneak in some little things before the candidate, then we freeze, right?
195 18:19:00 <stevenknight> yep
196 18:19:04 <garyo-home> good.
197 18:19:20 <stevenknight> okay, shall we move on to Jason_'s topic?
198 18:19:37 <garyo-home> yes.
199 18:19:57 <garyo-home> I looked a little at the doc for Parts; it is quite serious!
200 18:20:22 <Jason_> Well we have some new tool coming out... it has to be a bit serious
201 18:20:41 <garyo-home> Jason, I wonder if some of it couldn't be done with SCons Tools instead, but basically it looks useful.
202 18:20:49 <garyo-home> What's the new Intel tool coming out?
203 18:21:05 <Jason_> well I have a lot to say on all of this
204 18:21:21 <Jason_> Idea the doc i sent is about extending Scons...
205 18:21:31 <garyo-home> The basic idea is a way to make a scalable composition of libs etc., right?
206 18:21:32 <Jason_> here i think tools needs many improvements
207 18:22:05 <Jason_> well yes... but also about making large project sain to deal with
208 18:22:33 <garyo-home> Right, by imposing certain conventions that make it subdividable nicely.
209 18:22:36 <Jason_> having a unified build system is very important
210 18:22:55 <Jason_> but also trying to be flexible
211 18:23:07 <stevenknight> sorry, I haven't had time to look before now
212 18:23:09 <Jason_> it is a hard balance...
213 18:23:12 <garyo-home> Of course; lots of people have large unified build systems with SCons. Your way is one way of formalizing a structure.
214 18:23:18 <stevenknight> is the doc the .7z file you attached?
215 18:23:29 <garyo-home> Steven: yes, it's in there.
216 18:23:37 <Jason_> yes.. the problem with the team i work with is that we are all over the place
217 18:23:48 <Jason_> having a "make system" does not work well
218 18:23:56 <Jason_> yes
219 18:24:16 <stevenknight> agh, i don't have 7zip on this system
220 18:24:24 <Jason_> it is in word 2007 format.. I can give you an update in something else like html
221 18:25:25 <Jason_> so the first question for me is how can we best stare this code
222 18:25:54 <Jason_> My boss would like this to be part of SCons instead of it own project...
223 18:26:27 <Jason_> however it might be best to have it as a seperate project with heavy SCons involment
224 18:27:11 <garyo-home> I've long thought it would be good for SCons to have a contrib/ subtree, so that might be one way to go. The downside is you're tied to SCons release schedule and other stuff though. Separate is probably easier.
225 18:27:36 <Jason_> well we have our own svn branch here at Intel
226 18:27:48 <Jason_> ideally we would try to move to a public one
227 18:28:15 <Jason_> but we have some stuff in a subdir call "pieces" that i cannot ship
228 18:28:33 <garyo-home> Not sure what the best way to host a project like that is, but there are many places to look.
229 18:28:44 <Jason_> as it is internal add on that have knowleage of internal Intel networks
230 18:28:59 <garyo-home> You'll have to make it work "out of the box" from the public repo, of course.
231 18:29:10 <Jason_> yep that is the idea
232 18:29:22 <Jason_> the sample in the code i gave you should work out of the box
233 18:29:33 <garyo-home> I think everyone will be happier if it is separate from scons but closely related, at least for now.
234 18:29:37 <Jason_> the full sample will need svn to be installed :-)
235 18:30:09 <garyo-home> But we should spend some time at least reading the doc to get an idea of the scope and goals of the project.
236 18:30:17 <Jason_> If this agreed, I will start the process to make a tigris Parts project
237 18:30:39 <garyo-home> How about calling it SConsParts or something to show its relation to SCons?
238 18:30:58 <Jason_> that is fine with me
239 18:31:12 <Jason_> any other votes?
240 18:31:58 <garyo-home> Let's talk about the tech details and direction more at the next bug party meeting in 2 weeks.
241 18:32:11 <stevenknight> sorry, still trying to open things up
242 18:32:28 <stevenknight> got the .7z open, but i don't have Office on the system I switched to...
243 18:32:28 <Jason_> in 2 week I will be in florida *not* working
244 18:32:31 <garyo-home> btw, is this going to appear in a public tool from Intel?
245 18:32:35 <Jason_> so after that is fine
246 18:33:01 <garyo-home> Jason: ok, whenever you are around. But give us a week or so to look it over.
247 18:33:10 <Jason_> sure
248 18:33:14 <garyo-home> Esp. given 1.2 release schedule :-)
249 18:33:29 <stevenknight> Jason_: there's an early effort here at Google to do something that sounds similar to Parts
250 18:33:38 <stevenknight> but it looks like you're much farther along
251 18:33:46 <stevenknight> when I look, I
252 18:33:47 <Jason_> so the parts stuff will be public.. MIT under Intel with the intent to be move as much as possible to SCons
253 18:34:02 <Jason_> you work at Google
254 18:34:14 <stevenknight> I'm going to look for areas of possible cooperation
255 18:34:27 <stevenknight> yes, almost 11 months now
256 18:34:38 <Jason_> oh .. I thought it was VMware
257 18:34:45 <stevenknight> that was before, and on contract
258 18:34:56 <Jason_> well you should contact me... INtel and google have a lot going on
259 18:35:18 <stevenknight> the problem you're addressing is what we're running into too
260 18:35:46 <Jason_> so are many projects at Intel ....
261 18:35:56 <stevenknight> right, SCons is pretty good low-level plumbing
262 18:36:08 <Jason_> very powerful stuff however
263 18:36:15 <stevenknight> but there's no consistency for higher layer plug-and-play between components
264 18:36:35 <stevenknight> esp. in the open source world it all depends on how the person wrote the SCons config
265 18:36:37 <Jason_> I think as i under scons better and you parts.. I think there is a lot of value here to improve on
266 18:36:47 <stevenknight> agreed
267 18:37:10 <Jason_> the plug and play in very important
268 18:37:30 <stevenknight> right, it's the real value add of Autotools (IMHO)
269 18:37:31 <Jason_> when I was at a start up before intel.. one person controled the make scripts
270 18:37:58 <stevenknight> being able to approach any package and know how to at least build it consistently
271 18:38:12 <Jason_> in there larger cross geo.. developer fight over details and perfection
272 18:38:14 <stevenknight> sounds like Parts goes even further though
273 18:38:23 <Jason_> so everything breaks .
274 18:38:26 <stevenknight> right
275 18:38:32 <stevenknight> so following up on what Gary said
276 18:38:42 <Jason_> cross geo poltics i have found can make it hard to get simple stuff done
277 18:38:56 <stevenknight> I'm going to take a closer look after 1.2 is out
278 18:39:03 <Jason_> ok
279 18:39:24 <stevenknight> and perhaps touch base with you off line to compare first impressions
280 18:39:26 <Jason_> I will send out friday an update to the code and document ( small tweaks and fixes)
281 18:39:33 <stevenknight> thanks
282 18:39:41 <Jason_> so FYI
283 18:39:41 <stevenknight> if you can convert doc to some non-Word format that'd help me
284 18:39:51 <garyo-home> same here
285 18:39:49 <Jason_> WW 51 i am out of town
286 18:40:05 <garyo-home> What is "WW 51"?
287 18:40:08 <Jason_> I will be around other wise and will keep track of the e-mails
288 18:40:23 <Jason_> week of Dec 15
289 18:40:34 <garyo-home> ah yes.
290 18:40:43 <stevenknight> okay, i'll look for your update and we'll go from there
291 18:40:50 <Jason_> ok
292 18:40:57 <Jason_> thank you for your time
293 18:41:12 <garyo-home> ok, Steven I think it's your turn now!
294 18:41:17 <stevenknight> ok
295 18:41:21 <garyo-home> Jason_: yr welcome.
296 18:41:40 <stevenknight> one is the Visual C/C++ batch builder change I mentioned
297 18:41:58 <stevenknight> took a while, but it's close to solid
298 18:42:09 <stevenknight> some unit tests need attention
299 18:42:27 <garyo-home> Impressive, can't wait to try it. We have a lot of vc++ compiled into a few big libs.
300 18:42:54 <stevenknight> i'm going to try to get it in right away after 1.2 is out so it gets soak time
301 18:43:16 <stevenknight> it has some changes to the Taskmaster that aren't terribly extensive
302 18:43:23 <stevenknight> but may have unintended side effects
303 18:43:31 <GregoryNoel> (BRB. My wife just arrived with food and I've got to eat it while it's hot.)
304 18:43:56 <garyo-home> Do users have to do anything or does it just work?
305 18:43:59 <stevenknight> darn, bad timing, I was going to ask Greg something about Taskmaster NG
306 18:44:03 <Jason_> If you need someone to help test the taskmaster on a large project we are happy to help
307 18:44:24 <stevenknight> it's controlled by setting MSVC_BATCH=True (or 1 or whatever)
308 18:44:59 <garyo-home> sounds great. Does it generalize to other batch-buildable things?
309 18:45:06 <stevenknight> yes
310 18:45:10 <GregoryNoel> (I'm still reading, but I have to put down the burger to type.)
311 18:45:18 <stevenknight> okay, cool
312 18:45:21 <Jason_> if you can give me the detail of what to do i will give it a run on a few new i7 systems
313 18:45:41 <stevenknight> the Taskmaster issues is that this suggests the Taskmaster is handling the wrong granularity of input
314 18:45:45 <stevenknight> it's looking at individual Nodes
315 18:45:55 <stevenknight> and I'm beginning to think it should be looking at Executors
316 18:46:13 <stevenknight> GregoryNoel, just chew on that a little and let me know if it sounds crazy
317 18:46:16 <stevenknight> based on your TNG work
318 18:46:37 <stevenknight> you can do your own batch builder at the Action() level
319 18:46:51 <stevenknight> simplest case is Action('$FOOCOM', batch_key=True)
320 18:47:02 <GregoryNoel> That's what TNG does: the transverse graph is in (a wrapper around) the Executor.
321 18:47:14 <stevenknight> sweet, so I'm not entirely crazy, at least
322 18:47:21 <stevenknight> maybe this helps move us toward TNG
323 18:47:52 <stevenknight> batch_key=True will separate all calls to the given Builder using that action
324 18:48:59 <stevenknight> into grouped batches for the four-tuple (Action, env)
325 18:49:05 <stevenknight> excuse me, two-tuple
326 18:49:22 <stevenknight> batch_key can also be a function that takes (action, env, target, source)
327 18:49:39 <stevenknight> and returns the key to be used for separating batches
328 18:50:37 <stevenknight> so the MSVC_BATCH support uses the four-tuple (action, env, target.dir, source.dir) as the key
329 18:50:56 <garyo-home> That's very clever.
330 18:51:05 <stevenknight> with the upshot that you get batched compilation of all object files in a given directory
331 18:51:26 <stevenknight> because $CPPPPATH can affect your #includes based on the directory
332 18:51:36 <garyo-home> ... and the same compiler options etc.
333 18:51:49 <stevenknight> right, out of the same env
334 18:52:14 <garyo-home> Is it a lot faster?
335 18:52:31 <stevenknight> for our config (Google Chrome) only about 10%-15%
336 18:52:41 <stevenknight> which is nothing to sneeze at, but still, we were hoping for more
337 18:52:50 <garyo-home> That's nothing to sneeze at (jinx)
338 18:53:04 <stevenknight> on the other hand, i haven't gone in and profiled and optimizd it yet
339 18:53:17 <stevenknight> i'll probably upload to a subversion branch so people can try it out early
340 18:53:20 <garyo-home> For us, we use -O3 with the Intel compiler so all the time goes into the link step, but I'm sure this will help us even in that case.
341 18:53:21 <stevenknight> and send out email+doc
342 18:53:50 <stevenknight> (i've been doing this work experimenting with mercurial as a front end private branch development)
343 18:54:07 <stevenknight> okay, enough of that
344 18:54:24 <garyo-home> what else?
345 18:54:34 <stevenknight> next: Google contributors to SCons
346 18:54:41 <Jason_> so i have a question
347 18:54:49 <stevenknight> yes?
348 18:55:03 <Jason_> is there any plans to improve the depend checker for C/C++
349 18:55:16 <Jason_> there is a bit of an issue with BOOST headers for use
350 18:55:21 <stevenknight> i.e. make it like a real C preprocessor?
351 18:55:34 <Jason_> We have a work around... but it seem to cause more trouble than it solves
352 18:55:39 <stevenknight> there's an experimental module that will do that
353 18:56:08 <Jason_> well one that is smart enought to get the #include FOOBAR stuff
354 18:56:16 <stevenknight> yes
355 18:56:27 <stevenknight> take a look at (IIRC) Scanner/C.py
356 18:56:31 <Jason_> how can we try it out
357 18:56:59 <Jason_> branch in SCons?
358 18:57:01 <stevenknight> there's some commented out code that uses a cpp.py module to do enough "real" CPP stuff to handle boost
359 18:57:05 <Jason_> or in the main code
360 18:57:09 <stevenknight> no, it's checked in trunk
361 18:57:33 <stevenknight> just hasn't been productized by giving it the right user-configurable hook
362 18:57:43 <stevenknight> if you want to finish that piece and contribute it back, it'd be very welcome
363 18:58:04 <stevenknight> next: google contributors
364 18:58:30 <stevenknight> i have more people here starting to look at SCons internals
365 18:58:39 <stevenknight> especially w.r.t. performance
366 18:59:08 <stevenknight> which is good
367 18:59:23 <stevenknight> but we're finding that a lot of the flexibility we prize in SCons
368 18:59:53 <stevenknight> is antithetical to the kind of caching we'd need to do to really speed up things
369 19:00:08 <stevenknight> i'm wrestling with how best to channel this interest and energy
370 19:00:21 <garyo-home> that's hard.
371 19:00:34 <stevenknight> in ways that help SCons overall as well as Google's own purposes
372 19:00:50 <stevenknight> a Google fork of SCons is unattractive
373 19:00:58 <garyo-home> Right, certainly.
374 19:01:12 <garyo-home> Some kind of "mode"?
375 19:01:17 <stevenknight> but taking some of this work back into main line runs serious risk of breaking backwards compatibility
376 19:02:05 <garyo-home> and also making some existing idioms just plain not work, possibly.
377 19:02:12 <stevenknight> right
378 19:02:31 <garyo-home> How invasive is it?
379 19:02:57 <stevenknight> well, we're in early stages
380 19:03:07 <stevenknight> so more hypthetical than not
381 19:03:12 <garyo-home> ok.
382 19:03:27 <stevenknight> one patch i have queued tries to avoid starving child worker threads
383 19:03:39 <stevenknight> (actually doesn't try, it does avoid it)
384 19:03:57 <stevenknight> by just feeding as many ready Tasks as possible into the queue
385 19:04:00 <stevenknight> that doesn't break compatibility
386 19:04:20 <garyo-home> That seems great. In general, I think if it's really valuable, breaking simple back compat is ok (i.e. people have to do some simple recoding), but forcing larger changes is problematic.
387 19:04:23 <stevenknight> but it's in that pesky Taskmaster area where unintended affects abound
388 19:04:25 <GregoryNoel> Sounds like TNG
389 19:05:00 <stevenknight> mm, maybe I should send this patch to you to see what you think
390 19:05:36 <stevenknight> we were able to do some visualization (using Incredibuild) to see the starvation at work without the patch, and no starvation with it
391 19:06:32 <GregoryNoel> Sure, I can take a look. You should also look at TNG.
392 19:06:33 <stevenknight> tradeoff is more CPU cycles in the master thread searching the whole DAG for work more frequently
393 19:06:55 <GregoryNoel> TNG avoids that.
394 19:07:09 <stevenknight> okay, let's swap code and see how close we are
395 19:07:33 <stevenknight> are there issues holding up TNG?
396 19:08:02 <GregoryNoel> No, not really. Just lack of time.
397 19:08:08 <stevenknight> okay
398 19:08:12 <stevenknight> i'll send you the patch from here
399 19:08:37 <stevenknight> so now that I'm articulating this
400 19:08:38 <GregoryNoel> (here?)
401 19:08:43 <stevenknight> Google
402 19:09:30 <stevenknight> the meta issue is that this could bring a lot more impactive changes quickly into SCons
403 19:09:42 <stevenknight> with attendant possibility for destabilizing things
404 19:10:04 <stevenknight> in order to not hold up stuff, I'm going to have to spend (more) work time actually, oh, integrating patches
405 19:10:07 <GregoryNoel> (my fingers are still sticky, but I'm otherwise back.)
406 19:10:08 <stevenknight> in general, not just from Google
407 19:10:34 <stevenknight> and to do that I'm having to draft people here to cover some of the front-line work i've been trying to do on the Google Chrome build
408 19:11:12 <stevenknight> one other possibility I considered was to have a "google branch" in our SVN repository
409 19:11:29 <stevenknight> and pluck changes from that into trunk
410 19:11:32 <GregoryNoel> opposed: branches by features, not source
411 19:11:49 <stevenknight> yeah, that makes sense
412 19:12:23 <stevenknight> okay, so i just need to become johnny-on-the-spot with integration
413 19:12:33 <stevenknight> and maybe recruit additional committers / integrators
414 19:12:40 <stevenknight> from Google or elsewhere
415 19:12:50 <garyo-home> can you do that from Google? Would be useful.
416 19:13:05 <garyo-home> We haven't had much luck from the users list to date.
417 19:13:20 <stevenknight> can I do...? get more people from here working on SCons-the-project?
418 19:13:57 <garyo-home> Yes.
419 19:14:18 <stevenknight> there's certainly no institutional barrier
420 19:14:57 <garyo-home> Maybe get another googler or two on the dev list, tag along to a bug party...
421 19:15:01 <stevenknight> client-side projects here are getting behind SCons pretty heavily
422 19:15:09 <GregoryNoel> Doesn't Google have a rule that you can spend up to 20% of your time "doing good"? Recruit in that space.
423 19:15:43 <stevenknight> that still ends up being driven by whether people have an itch to scratch
424 19:15:51 <stevenknight> that they want to spend their 20% on
425 19:16:07 <garyo-home> but SCons is *sooo* *coool*!
426 19:16:07 <GregoryNoel> Of course. But fixing their build system to support themselves better is a strong itch.
427 19:16:08 <stevenknight> the potential itch here is speeding up builds for their projects
428 19:16:31 <garyo-home> That makes sense.
429 19:16:53 <Jason_> you need faster startup times?
430 19:17:01 <stevenknight> faster null build times
431 19:17:14 <garyo-home> steven: agreed, same here.
432 19:17:18 <GregoryNoel> ditto
433 19:17:24 <garyo-home> hence your caching ideas.
434 19:17:28 <Jason_> ya... big issue here.. I have a work around... but it a hack
435 19:17:37 <stevenknight> Jason_: what's your workaround?
436 19:17:48 <Jason_> we have a DB file we make
437 19:18:03 <Jason_> we use this to figure out if we need to define soemthing to SCons
438 19:18:13 <Jason_> this speeds up builds a great deal
439 19:18:26 <Jason_> we woudl have NULL build of 8-10 minutes
440 19:18:39 <stevenknight> so you only conditionally load parts of the configuration based on this DB?
441 19:18:47 <Jason_> got it down to 13-63
442 19:18:53 <Jason_> depending on the target
443 19:19:06 <Jason_> 13-63 secs
444 19:19:22 <Jason_> yes... but we have to have one good run
445 19:19:33 <stevenknight> right, to prep the DB with legitimate info
446 19:19:35 <stevenknight> that makes sense
447 19:19:39 <Jason_> else we don't know what is defined
448 19:20:01 <stevenknight> kind of related to this is making the DAG in SCons a real DAG
449 19:20:08 <stevenknight> (something I know Greg is very much in favor of)
450 19:20:18 <stevenknight> with separate tuples representing the edges
451 19:20:19 <Jason_> Same here
452 19:20:35 <GregoryNoel> Er, I thing you mean making the .sconsign a real DAG, but yes, I do.
453 19:20:42 <Jason_> then you can throw some well known task processing code at it
454 19:20:53 <Jason_> such as stuff used by a compiler
455 19:20:59 <stevenknight> right
456 19:20:59 <garyo-home> That scares me a little unless you explicitly allow runtime modification of the DAG
457 19:21:16 <GregoryNoel> white papers on request.
458 19:21:27 <stevenknight> and be able to infer what to build from known changed files
459 19:21:43 <garyo-home> ... but maybe users have to explicitly identify when they're modifying the DAG at runtime to invalidate and/or rescan
460 19:21:48 <stevenknight> instead of always having to walk from targets down, searching for what might have changed
461 19:21:57 <stevenknight> GregoryNoel: request, send white papers
462 19:22:19 <stevenknight> garyo-home: you have a use case in mind?
463 19:22:31 <garyo-home> Dynamic source generators.
464 19:22:42 <Jason_> Agreeded
465 19:22:46 <garyo-home> Where you don't know all the generated source names up front.
466 19:22:48 <Jason_> we have that as well
467 19:22:51 <stevenknight> that should still be possible
468 19:23:05 <garyo-home> Good, I rely heavily on that.
469 19:23:21 <stevenknight> i think the nodes would still have .sources, .implicit, .explicit etc.
470 19:23:23 <Jason_> a simple example if a Doxygen builder
471 19:23:31 <GregoryNoel> In point of fact, I'd like to optimize the null case first, when the SConscripts haven't been changed, and work from there.
472 19:23:33 <stevenknight> but instead of pointing directly to ohter Nodes
473 19:23:40 <Jason_> it can't really know what it will build up front
474 19:23:41 <stevenknight> they'd be lists of DAG edges
475 19:24:00 <stevenknight> GregoryNoel: agree in general
476 19:24:03 <garyo-home> Jason_: a very good use case there.
477 19:24:18 <stevenknight> but how do we know when the null case is optimized "enough" ?
478 19:24:30 <garyo-home> Steven: I have to think about that before I understand you.
479 19:24:34 <GregoryNoel> Zero time in parsing.
480 19:24:50 <stevenknight> wow, configurations are so different
481 19:25:00 <stevenknight> parsing is a really small part of our null build time
482 19:25:32 <GregoryNoel> I wonder...
483 19:26:02 <stevenknight> profiling shows we're heavily dominated by string (re-)expansion
484 19:26:46 <GregoryNoel> True, and working the bug list yesterday I found a case where a string was expanded *six* times to the same effect.
485 19:26:35 <Jason_> idea... you might want to look at the parts mapper.py file
486 19:26:59 <Jason_> we sort of replace expantions.. to get around the time issue here
487 19:27:07 <stevenknight> Jason_; aha
488 19:27:27 <stevenknight> I'll look at what you did as a source of ideas
489 19:27:40 <stevenknight> one of my big problems is I've spent so much time focusing on making corner cases work
490 19:27:44 <Jason_> part of this is me hacking a wrapper around the scanner object to force a few expansions
491 19:27:59 <stevenknight> that I'm prone to ignore effective solutions that really help the common case(s)
492 19:28:03 <Jason_> 5-10% difference in speed
493 19:28:32 <stevenknight> we got a pretty good improvement just running big long CPPPATH lists through env.subst() when assigning them
494 19:28:58 <Jason_> the problem was i did not see it when you added it :-(
495 19:29:13 <stevenknight> ? not in SCons itself, in our configuration
496 19:29:32 <Jason_> oh... never was able to use them
497 19:30:29 <stevenknight> anyone else about ready to turn into a pumpkin too?
498 19:30:36 <stevenknight> I should head for home...
499 19:30:39 <garyo-home> yes, time for me to go.
500 19:30:41 <GregoryNoel> Long past time for me
501 19:30:48 <stevenknight> okay, many thanks guys
502 19:30:52 <garyo-home> Thanks a lot as usual, guys!
503 19:30:54 <GregoryNoel> two weeks?
504 19:30:59 <garyo-home> ok for me.
505 19:31:00 <stevenknight> hopefully we'll see some increased activity coming up
506 19:31:06 <stevenknight> two weeks is good for me
507 19:31:07 <Jason_> latter!
508 19:31:12 <GregoryNoel> Next meeting 17 Dec then? 1.2 should be out, as well as the first checkpoint toward 1.3. Agreed?
509 19:31:12 <GregoryNoel> Next meeting 17 Dec then? 1.2 should be out, as well as the first checkpoint toward 1.3. Agreed?
510 19:31:12 <GregoryNoel> Next meeting 17 Dec then? 1.2 should be out, as well as the first checkpoint toward 1.3. Agreed?
511 19:31:13 <stevenknight> should have 1.2 out by then...
512 19:31:27 <GregoryNoel> oops...
513 19:31:26 <stevenknight> what I tell you three times is true...
514 19:31:29 <garyo-home> greg: yes, yes, yes.
515 19:32:04 <stevenknight> for the snark *was* a boojum, you see...
516 19:32:08 <garyo-home> :-)
517 19:32:17 <GregoryNoel> snicker-snack
518 19:32:17 <garyo-home> ok, g'night all.
519 19:32:22 * Jason_ has quit ("Leaving")
520 19:32:25 <GregoryNoel> cul
521 19:32:29 <stevenknight> later
522 19:32:39 * garyo-home has quit ("ChatZilla 0.9.84 [Firefox 3.0.4/2008102920]")
523 19:47:01 * stevenknight has quit ("Leaving")
524
