8.6b1.1 - failed to create a valid string rep

8.6b1.1 - failed to create a valid string rep

Post by neuronstor » Sun, 07 Jun 2009 07:02:26


Hi,

I've just checked out the current source of 8.6b1.1 and tried it on a
daemon I run on FreeBSD which has been running ok on 8.6a0


The process crashes with the following:

UpdateStringProc for type 'string' failed to create a valid string rep
Abort (core dumped)

It's a multithreaded app - so it might be a little tricky to find
exactly what triggers this.. any pointers?

Thanks,
Julian
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Don Porte » Sun, 07 Jun 2009 07:15:54


That's a new check that the contract of the UpdateStringProc()
is being met. It's possible your program has been silently failing
in this way all the time, and the only thing new is now you're being
informed about it.

Got any stack trace information at all?

How aggressively have you customized things? Can we assume the
type 'string' is the one provided by Tcl, or is it a custom replacement
in your program?



--
| Don Porter Mathematical and Computational Sciences Division |
| XXXX@XXXXX.COM Information Technology Laboratory |
| http://www.yqcomputer.com/ ~DPorter/ NIST |
|______________________________________________________________________|

 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Don Porte » Sun, 07 Jun 2009 07:27:15


Can you check whether the Tcl_Obj triggering this panic is an empty
string in "pure unicode" form? There's a gap in UpdateStringOfString
not checking for that case, but all the paths which should be able to
create it appear to have string rep creation covered.

It would be best to get this reported in the actual Tcl Bug Tracker
at Sourceforge, and include whatever you can about the value that
triggers the panic. Its contents and history.

--
| Don Porter Mathematical and Computational Sciences Division |
| XXXX@XXXXX.COM Information Technology Laboratory |
| http://www.yqcomputer.com/ ~DPorter/ NIST |
|______________________________________________________________________|
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by JMN » Sun, 07 Jun 2009 07:35:23


Hi Don,

It's a pure Tcl app - but uses a few extensions such as sqlite, tdom..
so no - I haven't done anything radical that I know of.
Sorry - no stack trace info - I'm not familiar with producing it.

Actually - my claim that it had been running ok wasn't quite true - as
it seemed to have a memory leak before and died after a few hours,
so perhaps that is related.

It appears to be occuring on an expression using 'ni' and a string
which was retrieved using tsv::get.
ie the program crashes on the 'if' line below.


set streams_on_thisthread [tsv::get t_
$allocated_tid accounts_on_thread]
if {"$accountid_with_pending/$logtype" ni
$streams_on_thisthread} {
tsv::lappend t_$allocated_tid
accounts_on_thread "$accountid_with_pending/$logtype"
}

I did also check out and build the thread extension at the same time.

JN
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by JMN » Sun, 07 Jun 2009 08:03:03


I'm not sure how to directly work out what's in the value at this
stage.
after the line:
set streams_on_thisthread [tsv::get t_$allocated_tid
accounts_on_thread]

it turns out that attempting to puts $streams_on_thisthread triggers
the crash.. which I guess isn't surprising.

As far as I can tell - the value should be an empty list at this stage
of the program's run.
The line where the value is set is:
tsv::array set t_$tid [list job_count 0 thread_started [clock
seconds] accounts_on_thread [list]]
I also tried:
tsv::array set t_$tid [list job_count 0 thread_started [clock
seconds] accounts_on_thread ""]

Interestingly.. by putting some junk in there - the crash is avoided.
e.g
tsv::array set t_$tid [list job_count 0 thread_started [clock
seconds] accounts_on_thread "spud"]

A space also works.


JN
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Alexandre » Sun, 07 Jun 2009 17:28:33


> seconds] accounts_on_thread [list]>
> I also tried>
> tsv::array set t_$tid [list job_count 0 thread_started [cl>ck
> seconds] accounts_on_thread >"]> >
> Interestingly.. y putting some junk in there - the crash is avoid>d.
> >.g
> tsv::array set t_$tid [list job_count 0 thread_started [>lock
> seconds] accounts_on_thread "s>ud>]
>
> A space also w>rk>.
>
> JN

I would try to look at the offending string object's internal fields,
eg with the Tweezer extension or similar (eg patch at
https://sourceforge.net/tracker/?func=detail&aid=1980917&group_id=10894&atid=310894
). For example:

- is the re>count >1 ?
- is the string rep lost at some time ?
- does the int rep change over time ?

-Alex
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by JMN » Sun, 07 Jun 2009 19:53:15

On Jun 6, 8:28m, Alexandre Ferrieux < XXXX@XXXXX.COM >




> > seconds] accounts_on_thread >"]> >
> > Interestingly.. y putting some junk in there - the crash is avoide>.>
> > >.>
> > tsv::array set t_$tid [list job_count 0 thread_started [>l>ck
> > seconds] accounts_on_thread "s>ud>]> >
> > A space also w>rk>.> >
>>> >N
>
> I would try to look at the offending string object's internal fi>lds,
> eg with the Tweezer extension or similar (eg patch athttps://sourceforge.net/tracker/?func=detail&aid=1980917&group_id=1>8...
> ). For exa>pl>:
>
> - is the >efcou>t >1 ?
> - is the string rep lost at so>e time ?
> - does the int rep change >ve> time ?
>
> -Alex- Hide qu>te> text -
>
> - Show quoted text -

Hmm.. not sure if I'm applying the patch correctly.
It said it succeeded, but make gives me:

In file included from /usr/local/buildtcl/8.6b1.1/tcl/generic/
regcustom.h:33,
from /usr/local/buildtcl/8.6b1.1/tcl/generic/
regguts.h:36,
from /usr/local/buildtcl/8.6b1.1/tcl/generic/
regcomp.c:33:
/usr/local/buildtcl/8.6b1.1/tcl/generic/tclInt.h:2756: error: storage
class specified for parameter `Tcl_DebugObjObjCmd'
/usr/local/buildtcl/8.6b1.1/tcl/generic/tclInt.h:2756: warning:
`__visibility__' attribute ignored
/usr/local/buildtcl/8.6b1.1/tcl/generic/tclInt.h:2752: error:
parameter "interp" has just a forward declaration
/usr/local/buildtcl/8.6b1.1/tcl/generic/tclInt.h:2752: error:
parameter "listPtr" has just a forward declaration
/usr/local/buildtcl/8.6b1.1/tcl/generic/tclInt.h:2756: error:
parameter "Tcl_DebugObjObjCmd" has just a forward declaration
*** Error code 1

JN
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by JMN » Mon, 08 Jun 2009 02:12:55

On Jun 6, 8:28m, Alexandre Ferrieux < XXXX@XXXXX.COM >




> > seconds] accounts_on_thread >"]> >
> > Interestingly.. y putting some junk in there - the crash is avoide>.>
> > >.>
> > tsv::array set t_$tid [list job_count 0 thread_started [>l>ck
> > seconds] accounts_on_thread "s>ud>]> >
> > A space also w>rk>.> >
>>> >N
>
> I would try to look at the offending string object's internal fi>lds,
> eg with the Tweezer extension or similar (eg patch athttps://sourceforge.net/tracker/?func=detail&aid=1980917&group_id=1>8...
> ). For exa>pl>:
>
> - is the >efcou>t >1 ?
> - is the string rep lost at so>e time ?
> - does the int rep change >ve> time ?
>
> -Alex- Hide qu>te> text -
>
> - Show quoted text -

tweezer type: string
tweezer refcount: 2
string length: 0

string bytelength triggers the crash.
Attempting to tweezer shimmer the value to another type such as list
or bytearray also triggers the crash.

Nothing has been written to that element of the array since it was
initially setup using:
tsv::set t_$tid accounts_on_thread [list]

(I split up the previous 'tsv::array set' to try to isolate it)

tsv::set t_$tid accounts_on_thread ""

also has the problem

tsv::set t_$tid accounts_on_thread " "

Is a workaround that fixes my app.

I've been unable to produce a simple script which demonstrates the
crash.

JN
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Alexandre » Mon, 08 Jun 2009 19:18:50

n Jun 6, 12:53m, JMN < XXXX@XXXXX.COM > wrote:
> > > seconds] accounts_on_thread "s>ud>]> > > > A space also w>rk>.> > >>> >N> >
> > I would try to look at the offending string object's internal fi>l>s,
> > eg with the Tweezer extension or similar (eg patch athttps://sourceforge.net/tracker/?func=detail&aid=1980917&group_id=1>8>..
> > ). For exa>pl>:> >
> > - is the >efcou>t>>1 ?
> > - is the string rep lost at so>e>time ?
> > - does the int rep change >ve> >ime ?
>
> > -Alex- Hide qu>te> >ext -
>
> > - Show qu>te> text -
>
> Hmm.. not sure if I'm applying the patch>correctly.
> It said it succeeded, but mak> g>ves me:
>
> In file included from /usr/local/buildtcl/8.6b1.1/t>l/generic/
> regc>stom.h:33,
> rom /usr/local/buildtcl/8>6b1.1/tcl/generi>/
> regguts.h:36,
> rom /usr/local/bu>ldtcl/8.6b1.1/tc>/generic/
> regcomp.c:33:
> /usr/local/buildtcl/8.6b1.1/tcl/generic/tc>Int.h:2756: error: storage
> class specified for par>meter `Tcl_DebugObjObjCmd'
> /usr/local/buildtcl/8.6b1.1/tcl/gene>ic/tclInt.h:2756: warning:
> `__visi>ility__' attribute ignored
> /usr/local/buildtcl/8.6b1.1/tcl/ge>eric/tclInt.h:2752: error:
> parameter "interp" has>just a forward declaration
> /usr/local/buildtcl/8.6b1.1/tcl/ge>eric/tclInt.h:2752: error:
> parameter "listPtr" has>just a forward declaration
> /usr/local/buildtcl/8.6b1.1/tcl/ge>eric/tclInt.h:2756: error:
> parameter "Tcl_DebugObjObjCmd" has>just a forward declaration
> *** Error code 1

Oops, I realize this was a fragile diff. I have now updated it to a
more robust one (cvs diff -u), relative to the current HEAD. Thanks
for the feedback.

-Alex
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Alexandre » Mon, 08 Jun 2009 19:24:14

n Jun 6, 7:12m, JMN < XXXX@XXXXX.COM > wrote:
> > > seconds] accounts_on_thread "s>ud>]> > > > A space also w>rk>.> > >>> >N> >
> > I would try to look at the offending string object's internal fi>l>s,
> > eg with the Tweezer extension or similar (eg patch athttps://sourceforge.net/tracker/?func=detail&aid=1980917&group_id=1>8>..
> > ). For exa>pl>:> >
> > - is the >efcou>t>>1 ?
> > - is the string rep lost at so>e>time ?
> > - does the int rep change >ve> >ime ?
>
> > -Alex- Hide qu>te> >ext -
>
> > - Show qu>te> text -
>
> tweezer ty>e: tring
> tweezer >efcount: 2
> strin> l>ngth: 0
>
> string bytelength triggers>the crash.
> Attempting to tweezer shimmer the value to another type s>ch as list
> or bytearray also triggers>th> crash.
>
> Nothing has been written to that element of the array s>nce it was
> initially s>tup using:
> tsv::set t_$tid accounts_on_th>ea> [list]
>
> (I split up the previous 'tsv::array set' to try to >so>ate it)
>
> sv::set t_$tid accounts_>n_>hread ""
>
> also has>th> problem
>
> tsv::set t_$tid accounts_>n_>hread " "
>
> Is a workaround that >ix>s my app.
>
> I've been unable to produce a simple script which dem>nstrates >he> > crash.
>
> JN

The main difference between "" and " " is that "" is likely to be a
more heavily shared literal.
So I would predict that obtaining a fresh, unshared empty string by
replacing "" with (e.g.) [list], would avoid the bug too.

Now please follow Don's advice and open a bug ticket like "TSV
misbehaves with shared literals". We'll follow up from there.

-Alex


 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Bruce Hart » Tue, 09 Jun 2009 22:30:08

lexandre Ferrieux wrote:

except that is what his original code was doing in the first place


definitely file the bug on tsv behaviors, but I would jump to the conclusion
that it is only shared literals only.

Bruce
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Alexandre » Wed, 10 Jun 2009 01:28:06

n Jun 8, 3:30m, Bruce Hartweg < XXXX@XXXXX.COM > wrote:
> >>>> seconds] accounts_on_thread "s>u>>>> > >>>> A space also w>r>>>> > >>>>>>>
> >>> I would try to look at the offending string object's internal fi>l>>>
> >>> eg with the Tweezer extension or similar (eg patch athttps://sourceforge.net/tracker/?func=detail&aid=1980917&group_id=1>8>>>
> >>> ). For exa>p>>>
> >>> - is the >efcou>t>>> ?
> >>> - is the string rep lost at so>e>>>me ?
> >>> - does the int rep change >v>>>time ?
> >>> -Alex- Hide qu>t>>>text -
> >>> - Show qu>t>> text -
> >> tweezer ty>e>>tring
> >> tweezer >e>>ount: 2
> >> strin> l>n>>h: 0
>
> >> string bytelength triggers>t>> crash.
> >> Attempting to tweezer shimmer the value to another type s>c>>as list
> >> or bytearray also triggers>th> >>ash.
>
> >> Nothing has been written to that element of the array s>n>> it was
> >> initially s>t>> using:
> >> tsv::set t_$tid accounts_on_th>ea> >>ist]
>
> >> (I split up the previous 'tsv::array set' to try to >so>a>> it)
>
> >> sv::set t_$tid accounts_>n_>h>>ad ""
>
> >> also has>th> >>oblem
>
> >> tsv::set t_$tid accounts_>n_>h>>ad " "
>
> >> Is a workaround that >ix>s>>y app.
>
> >> I've been unable to produce a simple script which dem>n>>rates the> >>>>>crash>
>
> > The main difference between "" and " " is that "" is l>k>ly to be a
> > more heavily sh>r>d literal.
> > So I would predict that obtaining a fresh, unshared em>t> string by
> > replacing "" with (e.g.) [list], would avoid>th> bug too.
>
> except that is what his original code was doing in the first place > >Oops :-}

> > Now please follow Don's advice and open a bug tic>e> like "TSV
> > misbehaves with shared literals". We'll follow u> f>om there.
>
> definitely file the bug on tsv behaviors, but I would jump to t>e conclusion
> that it is only shared literals only.

However, further investigation shows that [list] and "" in procs get
compiled to the same, shared literal.
Try [string range a 0 -1], just tested, it breaks its ties to the
literal pool.

-Alex
 
 
 

8.6b1.1 - failed to create a valid string rep

Post by JMN » Wed, 10 Jun 2009 03:20:04

n Jun 8, 4:28m, Alexandre Ferrieux < XXXX@XXXXX.COM >
wrote:
> > >>> - is the string rep lost at so>e>t>>> ?
> > >>> - does the int rep change >v>r>>>me ?
> > >>> -Alex- Hide qu>t>d>>>xt -
> > >>> - Show qu>t>d>>ext -
> > >> tweezer ty>e> >>tring
> > >> tweezer >e>c>>nt: 2
> > >> strin> l>n>t>> 0
>
> > >> string bytelength triggers>t>e>>rash.
> > >> Attempting to tweezer shimmer the value to another type s>c> >> list
> > >> or bytearray also triggers>th> >r>>h.
>
> > >> Nothing has been written to that element of the array s>n>e>>t was
> > >> initially s>t>p>>sing:
> > >> tsv::set t_$tid accounts_on_th>ea> >l>>t]
>
> > >> (I split up the previous 'tsv::array set' to try to >so>a>e>>t)
>
> > >> sv::set t_$tid accounts_>n_>h>e>> ""
>
> > >> also has>th> >r>>lem
>
> > >> tsv::set t_$tid accounts_>n_>h>e>> " "
>
> > >> Is a workaround that >ix>s>m>>app.
>
> > >> I've been unable to produce a simple script which dem>n>t>>tes the
>
> > > The main difference between "" and " " is that "" is l>k>l> to be a
> > > more heavily sh>r>d>literal.
> > > So I would predict that obtaining a fresh, unshared em>t> >tring by
> > > replacing "" with (e.g.) [list], would avoid>th> >ug too.
>
> > except that is what his original code was doing in th> f>rst place
>
> > > Now please follow Don's advice and open a bug tic>e> >ike "TSV
> > > misbehaves with shared literals". We'll follow u> f>o> there.
>
> > definitely file the bug on tsv behaviors, but I would jump to t>e>conclusion
> > that it is only shared l>te>als only.
>
> However, further investigation shows that [list] and "">in procs get
> compiled to the same, sh>red literal.
> Try [string range a 0 -1], just tested, it breaks it> ties to the
> >it>ral pool.
>
> -Alex

Thanks.. tried your suggesion of [string range a 0 -1] and the bug
does not occur with that value.
It's also absent with [string range "" 0 0]

Bug ticket # 2803109 opened.

JN


 
 
 

8.6b1.1 - failed to create a valid string rep

Post by Alexandre » Wed, 10 Jun 2009 07:11:18


A quick update: Don and Joe (Mistachkin) nailed it down. It was/is a
bad interaction between questionable intrep surgery in the Thread
extension, and a recent, unwanted restriction on UpdateStringOfString
in the core. Patch including test case in the oven.

-Alex