How do you set the precision of a Real without converting to a String?

Is there a way to set the precision of a Real without having to convert it to string and then back.

For example
var myNumber = 3.23323424234
var result = toReal(format(“{0,number,######.##}”, myNumber))

will give me back 3.23. It seems odd I would have to do it this way.  While toDecimal  has precision, this won’t work for me because of its String representation and what I’m trying to do.

Am I stuck with this conversion?

  • This topic was modified 3 years, 7 months ago by Ken.
Posted: July 31, 2020 at 4:33 pm
Replies: 2
Jul 31, 2020
Posted by Ken

Real is a pure 64-bit floating point value (Java double), so it does not have a precision.  That is what Decimal is for.  What about its string representation doesn’t work?

It may look like you get back 3.23, but since this is a binary floating point representation, that may not be what you really have.  For example:

(0.1 + 0.2 == 0.3)

Actually evaluates to false, not true.

Jul 31, 2020
Posted by dsprinzen

There is nothing wrong with the string representation inside the platform.  But I need the resulting JSON object that I am going to plug the number into to not have quotes around it.

So if I did something like this:

var number = toDecimal((randomFloat() * (max – min) + min ), 2)
return {val:number}

I might get back something like

{
“val”: “24.41”
}

If I send it to another service or if I called it via an API externally, I get back a string.`

  • This reply was modified 3 years, 7 months ago by dsprinzen.
  • This reply was modified 3 years, 7 months ago by dsprinzen.
  • This reply was modified 3 years, 7 months ago by dsprinzen.
  • This reply was modified 3 years, 7 months ago by dsprinzen.
Viewing 3 posts - 1 through 3 (of 3 total)
  • You must be logged in to reply to this topic.
© Vantiq 2024 All rights reserved  •  Vantiq Corporation’s Privacy Policy and Cookie Policy