set int64 gives a wrong value

Coldzer0 commented

2018-08-17 21:39:36 +00:00

(Migrated from github.com)

Hello BeRo
while i'm testing setting large values like
18374966859414961920

i test with

var x = 18374966859414961920
undefined
x
18374966859414962000

i try to follow the code and i think the problem is here

DoFastShortest() in BESENNumberUtils

i try to fix but i can't

can you please check it if you have time

thanks

Hello BeRo while i'm testing setting large values like 18374966859414961920 i test with >var x = 18374966859414961920 undefined >x 18374966859414962000 > i try to follow the code and i think the problem is here DoFastShortest() in BESENNumberUtils i try to fix but i can't can you please check it if you have time thanks

Coldzer0 commented

2018-08-17 21:49:58 +00:00

(Migrated from github.com)

i think there's no problem with your code
i'm sorry for judge so fast
i test on V8 Engine and give same result

but i don't know why or how to set to the value i want

i think there's no problem with your code i'm sorry for judge so fast i test on V8 Engine and give same result but i don't know why or how to set to the value i want

BeRo1985 commented

2018-08-17 23:53:14 +00:00

(Migrated from github.com)

(Mostly) all number values at ECMAScript/JavaScript are IEEE754 64-bit Double-Precision Floating Point values, and these can hold only up to 53-bit integer values lossless. See http://2ality.com/2012/04/number-encoding.html and https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER and so on. So it's here by language-design.

👍 1

Coldzer0 commented

2018-08-18 01:34:57 +00:00

(Migrated from github.com)

@BeRo1985 do you plan to update to ES6 ? or update the current project !

Rows
Columns

set int64 gives a wrong value #15